EngineeringException

another way of creating RAM disc in Mac

2012-02-18T21:38:00.000+09:00

I wrote about a way of creating RAM disc in Mac. But I knew that using discutil is better than mount for creating RAM disc in Mac.

Create a RAM Disk in Mac OS X

$ diskutil erasevolume HFS+ "ramdisk" `hdiutil attach -nomount ram://1165430`
Started erase on disk1
Unmounting disk
Erasing
Initialized /dev/rdisk1 as a 569 MB HFS Plus volume
Mounting disk
Finished erase on disk1 ramdisk

The good thing is the disc is visible on Finder. Of course, the disk free space can be checked easily.

Thanks.

af* command for treating audio files in Mac

2012-02-11T17:06:00.001+09:00

I knew that there are commands for treating audio files in Mac by default.

afinfo

afinfo is a command for showing information of audio file.

$ afinfo happykids.wav 
File:           happykids.wav
File type ID:   WAVE
Data format:     1 ch,  11025 Hz, 'lpcm' (0x00000008) 8-bit unsigned integer
                no channel layout.
estimated duration: 5.576417 sec
audio bytes: 61480
audio packets: 61480
bit rate: 88200 bits per second
packet size upper bound: 1
audio data file offset: 44
optimized
source bit depth: I8
----

afconvert

afconvert is a command for converting format of audio file. The supported formats are showed by option: -hf

$ afconvert -hf
Audio file and data formats:
    '3gpp' = 3GP Audio (.3gp)
               data_formats: 'Qclp' 'aac ' 'aace' 'aach' 'aacl' 'aacp' 
                             'samr' 
    '3gp2' = 3GPP-2 Audio (.3g2)
               data_formats: 'Qclp' 'aac ' 'aace' 'aach' 'aacl' 'aacp' 
                             'samr' 
    'adts' = AAC ADTS (.aac, .adts)
               data_formats: 'aac ' 'aach' 'aacp' 
    'ac-3' = AC3 (.ac3)
               data_formats: 'ac-3' 
    'AIFC' = AIFC (.aifc, .aiff, .aif)
               data_formats: I8 BEI16 BEI24 BEI32 BEF32 BEF64 UI8 'ulaw' 
                             'alaw' 'MAC3' 'MAC6' 'ima4' 'QDMC' 'QDM2' 
                             'Qclp' 'agsm' 
    'AIFF' = AIFF (.aiff, .aif)
               data_formats: I8 BEI16 BEI24 BEI32 
    'amrf' = AMR (.amr)
               data_formats: 'samr' 
    'm4af' = Apple MPEG-4 Audio (.m4a, .m4r)
               data_formats: 'aac ' 'aace' 'aach' 'aacl' 'aacp' 'alac' 
    'caff' = CAF (.caf)
               data_formats: '.mp1' '.mp2' '.mp3' 'QDM2' 'QDMC' 'Qclp' 
                             'Qclq' 'aac ' 'aace' 'aach' 'aacl' 'aacp' 
                             'alac' 'alaw' 'dvi8' 'ilbc' 'ima4' I8 BEI16 
                             BEI24 BEI32 BEF32 BEF64 LEI16 LEI24 LEI32 
                             LEF32 LEF64 'ms\x00\x02' 'ms\x00\x11' 'ms\x001' 
                             'paac' 'samr' 'ulaw' 
    'MPG1' = MPEG Layer 1 (.mp1, .mpeg, .mpa)
               data_formats: '.mp1' 
    'MPG2' = MPEG Layer 2 (.mp2, .mpeg, .mpa)
               data_formats: '.mp2' 
    'MPG3' = MPEG Layer 3 (.mp3, .mpeg, .mpa)
               data_formats: '.mp3' 
    'mp4f' = MPEG-4 Audio (.mp4)
               data_formats: 'aac ' 'aace' 'aach' 'aacl' 'aacp' 
    'NeXT' = NeXT/Sun (.snd, .au)
               data_formats: I8 BEI16 BEI24 BEI32 BEF32 BEF64 'ulaw' 
    'Sd2f' = Sound Designer II (.sd2)
               data_formats: I8 BEI16 BEI24 BEI32 
    'WAVE' = WAVE (.wav)
               data_formats: UI8 LEI16 LEI24 LEI32 LEF32 LEF64 'ulaw' 
                             'alaw'

There is "MPG3" in a list of supported formats, but Mac doesn't provide any MP3 encoder.

$ afconvert -f MPG3 -d '.mp3' happykids.wav 
Error: ExtAudioFileSetProperty ('cfmt') failed ('fmt?')

$ afconvert -f m4af -d aac happykids.wav

$ afinfo happykids.m4a 
File:           happykids.m4a
File type ID:   m4af
Data format:     1 ch,  11025 Hz, 'aac ' (0x00000000) 0 bits/channel, 0 bytes/packet, 1024 frames/packet, 0 bytes/frame
Channel layout: Mono
estimated duration: 5.576417 sec
audio bytes: 15950
audio packets: 63
audio 61480 valid frames + 2112 priming + 920 remainder = 64512
bit rate: 21806 bits per second
packet size upper bound: 399
audio data file offset: 4096
optimized
format list:
[ 0] format:     1 ch,  11025 Hz, 'aac ' (0x00000000) 0 bits/channel, 0 bytes/packet, 1024 frames/packet, 0 bytes/frame
     Channel layout: Mono
source bit depth: I8
----

afplay

afplay is a command for playback of audio file. Probably, the supported formats are same as afconvert.

I found some weblogs which said that the sound quality of afplay is better than iTunes. But I can't tell the difference.

Thanks.

Removing margins of PDF generated by iBooks Author

2012-01-30T22:20:00.000+09:00

iBooks Author is very solicitous and beautiful Desktop publishing tool. But I have a complaint about this. It is that margins and a watermark break into PDF generated by iBooks Author.

Probably, Apple indicate PDF generated by iBooks Author should not be published because they think that the function of exporting PDF should be used for confirming the content and layout on Mac when we don't have iPad accidentally.

But, I want to make ebooks with iBooks Author and publish them in good-looking and widely-used PDF on the Web, too.

I tried to remove margins from PDF generated iBooks Author in two ways.

pdfcrop

pdfcrop is a software for removing margins from PDF. This is supplied with TeX Live.

pdfcrop automatically recognizes margins in PDF. But pdfcrop without options yields a disappointing result from PDF genetated by iBooks Author because there is a watermark " iBooks Author" in the bottom of it. Therefore, we need to add options for specifying the size of the bottom margin.

$ time pdfcrop --margins "0 0 0 -55" ibooks_pdf.pdf ibooks_pdf_cropped.pdf
PDFCROP 1.32, 2011/08/10 - Copyright (c) 2002-2011 by Heiko Oberdiek.
==> 4 pages written on `ibooks_pdf_cropped.pdf'.

real    2m51.121s
user    2m43.072s
sys     0m0.714s

It took a few minutes to remove margins of 4 pages of PDF.

$ identify ibooks_pdf.pdf
ibooks_pdf.pdf[0] PDF 1024x748 1024x748+0+0 16-bit Bilevel DirectClass 95.8KB 0.010u 0:00.009
ibooks_pdf.pdf[1] PDF 1024x748 1024x748+0+0 16-bit Bilevel DirectClass 95.8KB 0.000u 0:00.009
ibooks_pdf.pdf[2] PDF 1024x748 1024x748+0+0 16-bit Bilevel DirectClass 95.8KB 0.000u 0:00.009
ibooks_pdf.pdf[3] PDF 1024x748 1024x748+0+0 16-bit Bilevel DirectClass 95.8KB 0.000u 0:00.009

$ identify ibooks_pdf_cropped.pdf
ibooks_pdf_cropped.pdf[0] PDF 922x598 922x598+0+0 16-bit Bilevel DirectClass 69.4KB 0.000u 0:00.019
ibooks_pdf_cropped.pdf[1] PDF 922x598 922x598+0+0 16-bit Bilevel DirectClass 69.4KB 0.000u 0:00.019
ibooks_pdf_cropped.pdf[2] PDF 922x598 922x598+0+0 16-bit Bilevel DirectClass 69.4KB 0.000u 0:00.019
ibooks_pdf_cropped.pdf[3] PDF 922x598 922x598+0+0 16-bit Bilevel DirectClass 69.4KB 0.000u 0:00.019

The result is good, but character information of all pages are lost. Thus, we can't select text on the processed PDF because all pages are imaged.

PDF hack

PDF can have information for the paper size (MediaBox) and the display size (CropBox). There is MediaBox in PDF generated by iBooks Author, but there is not CropBox. I added CropBox into PDF with shell command. The size was decided by the result of being processed by pdfcrop.

$ time sed -e 's;/MediaBox;/CropBox [52 75 973 673] /MediaBox;g' ibooks_pdf.pdf > ibooks_pdf_cropped.pdf

real    0m0.004s
user    0m0.002s
sys     0m0.003s

The process was complete in a moment. I confirmed that the processed PDF can be rendered with Preview.app and Google Chrome (Chrome PDF Viewer plugin). But it seems that some application can't reder the processed PDF.

$ identify ibooks_pdf_cropped.pdf 
   **** Warning:  An error occurred while reading an XREF table.
   **** The file has been damaged.  This may have been caused
   **** by a problem while converting or transfering the file.
   **** Ghostscript will attempt to recover the data.

   **** This file had errors that were repaired or ignored.
   **** The file was produced by: 
   **** >>>> Mac OS X 10.7.2 Quartz PDFContext <<<<
   **** Please notify the author of the software that produced this
   **** file that it does not conform to Adobe's published PDF
   **** specification.

ibooks_pdf_cropped.pdf[0] PDF 1024x748 1024x748+0+0 16-bit Bilevel DirectClass 95.8KB 0.010u 0:00.019
ibooks_pdf_cropped.pdf[1] PDF 1024x748 1024x748+0+0 16-bit Bilevel DirectClass 95.8KB 0.000u 0:00.019
ibooks_pdf_cropped.pdf[2] PDF 1024x748 1024x748+0+0 16-bit Bilevel DirectClass 95.8KB 0.000u 0:00.009
ibooks_pdf_cropped.pdf[3] PDF 1024x748 1024x748+0+0 16-bit Bilevel DirectClass 95.8KB 0.000u 0:00.009

I wrote AppleScript droplet and created the project for simplifying the process to crop PDF by sed.

laclefyoshi / cropping_ibooksauthor_pdf / overview — Bitbucket

Thanks.

Outputting data for people with tablib

2012-01-21T12:44:00.000+09:00

In order to show some data in processing to others (that are boss or client in business), I think it tiresome to copy and paste data into Excel sheets or export HTML tables.

I found tablib that is a library for formatting data as tabular form or semistructured form.

== show attributes of Dataset ==
['_Dataset__headers', '__class__', '__delattr__', '__delitem__', '__dict__', '__doc__', '__format__', '__getattribute__', '__getitem__', '__hash__', '__init__', '__len__', '__module__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__setitem__', '__sizeof__', '__str__', '__subclasshook__', '__weakref__', '_clean_col', '_data', '_formatters', '_get_dict', '_get_headers', '_package', '_register_formats', '_separators', '_set_dict', '_set_headers', '_validate', 'add_formatter', 'append', 'append_col', 'append_separator', 'csv', 'dict', 'filter', 'headers', 'height', 'html', 'insert', 'insert_col', 'insert_separator', 'json', 'lpop', 'lpush', 'lpush_col', 'ods', 'pop', 'rpop', 'rpush', 'rpush_col', 'sort', 'stack', 'stack_cols', 'title', 'transpose', 'tsv', 'width', 'wipe', 'xls', 'xlsx', 'yaml']

== output as dict ==
[OrderedDict([('name', 'Alice'), ('age', 21)]), OrderedDict([('name', 'Bob'), ('age', 20)]), OrderedDict([('name', 'Carol'), ('age', 31)]), OrderedDict([('name', 'Taro'), ('age', 10)])]

== output as json ==
[{"name": "Alice", "age": 21}, {"name": "Bob", "age": 20}, {"name": "Carol", "age": 31}, {"name": "Taro", "age": 10}]

== output only female as json ==
[{"name": "Alice", "age": 21}, {"name": "Carol", "age": 31}]

== output as csv ==
name,age,e/o
Alice,21,odd
Bob,20,even
Carol,31,odd
Taro,10,even


== show attributes of Databook ==
['__class__', '__delattr__', '__dict__', '__doc__', '__format__', '__getattribute__', '__hash__', '__init__', '__module__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '__weakref__', '_datasets', '_package', '_register_formats', 'add_sheet', 'html', 'json', 'ods', 'size', 'wipe', 'xls', 'xlsx', 'yaml']

== output 2 sheets as yaml ==
- !!python/object/apply:collections.OrderedDict
  - - - data
      - - !!python/object/apply:collections.OrderedDict
          - - [name, Alice]
            - [age, 21]
            - [e/o, odd]
        - !!python/object/apply:collections.OrderedDict
          - - [name, Bob]
            - [age, 20]
            - [e/o, even]
        - !!python/object/apply:collections.OrderedDict
          - - [name, Carol]
            - [age, 31]
            - [e/o, odd]
        - !!python/object/apply:collections.OrderedDict
          - - [name, Taro]
            - [age, 10]
            - [e/o, even]
    - [title, null]
- !!python/object/apply:collections.OrderedDict
  - - - data
      - - !!python/object/apply:collections.OrderedDict
          - - [name, Alice]
            - [age, 21]
            - [e/o, odd]
        - !!python/object/apply:collections.OrderedDict
          - - [name, Bob]
            - [age, 20]
            - [e/o, even]
        - !!python/object/apply:collections.OrderedDict
          - - [name, Carol]
            - [age, 31]
            - [e/o, odd]
        - !!python/object/apply:collections.OrderedDict
          - - [name, Taro]
            - [age, 10]
            - [e/o, even]
    - [title, null]


== output 2 sheets as html ==
<h3>Set 0</h3>
<table>
<thead>
<tr><th>name</th>
<th>age</th>
<th>e/o</th></tr>
</thead>
<tr><td>Alice</td>
<td>21</td>
<td>odd</td></tr>
<tr><td>Bob</td>
<td>20</td>
<td>even</td></tr>
<tr><td>Carol</td>
<td>31</td>
<td>odd</td></tr>
<tr><td>Taro</td>
<td>10</td>
<td>even</td></tr>
</table>
<h3>Set 1</h3>
<table>
<thead>
<tr><th>name</th>
<th>age</th>
<th>e/o</th></tr>
</thead>
<tr><td>Alice</td>
<td>21</td>
<td>odd</td></tr>
<tr><td>Bob</td>
<td>20</td>
<td>even</td></tr>
<tr><td>Carol</td>
<td>31</td>
<td>odd</td></tr>
<tr><td>Taro</td>
<td>10</td>
<td>even</td></tr>
</table>

Unfortunately, tablib can't be used in Jython. According to error message, this problem is caused by grammer.py in odfpy.

$ jython
Jython 2.5.2 (Release_2_5_2:7206, Mar 2 2011, 23:12:06)
[Java HotSpot(TM) 64-Bit Server VM (Apple Inc.)] on java1.6.0_29
Type "help", "copyright", "credits" or "license" for more information.
>>> import tablib
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "tablib/__init__.py", line 3, in <module>
    from tablib.core import (
  File "tablib/core.py", line 15, in <module>
    from tablib import formats
  File "tablib/formats/__init__.py", line 6, in <module>
    from . import _csv as csv
  File "tablib/formats/_csv.py", line 6, in <module>
    from tablib.compat import is_py3, csv, StringIO
  File "tablib/compat.py", line 45, in <module>
    from tablib.packages.odf import opendocument, style, text, table
  File "tablib/packages/odf/opendocument.py", line 26, in <module>
    import manifest, meta
  File "tablib/packages/odf/manifest.py", line 24, in <module>
    from element import Element
  File "tablib/packages/odf/element.py", line 28, in <module>
    import grammar
java.lang.ClassFormatError: Invalid method Code length 370403 in class file tablib/packages/odf/grammar$py
 at java.lang.ClassLoader.defineClass1(Native Method)
 at java.lang.ClassLoader.defineClassCond(ClassLoader.java:631)
 at java.lang.ClassLoader.defineClass(ClassLoader.java:615)
 at org.python.core.BytecodeLoader$Loader.loadClassFromBytes(BytecodeLoader.java:119)

...

java.lang.ClassFormatError: java.lang.ClassFormatError: Invalid method Code length 370403 in class file tablib/packages/odf/grammar$py

When I disabled importing pyodf, tablib can be used in Jython.

$ python
>>> import tablib
>>> print tablib.formats.available
(<module 'tablib.formats._json' from 'tablib/formats/_json.py'>, <module 'tablib.formats._xls' from 'tablib/formats/_xls.py'>, <module 'tablib.formats._yaml' from 'tablib/formats/_yaml.py'>, <module 'tablib.formats._csv' from 'tablib/formats/_csv.py'>, <module 'tablib.formats._tsv' from 'tablib/formats/_tsv.py'>, <module 'tablib.formats._html' from 'tablib/formats/_html.py'>, <module 'tablib.formats._xlsx' from 'tablib/formats/_xlsx.py'>, <module 'tablib.formats._ods' from 'tablib/formats/_ods.py'>)

$ jython
>>> import tablib
>>> print tablib.formats.available
(<module 'tablib.formats._json' from 'tablib/formats/_json.py'>, <module 'tablib.formats._xls' from 'tablib/formats/_xls.py'>, <module 'tablib.formats._yaml' from 'tablib/formats/_yaml.py'>, <module 'tablib.formats._csv' from 'tablib/formats/_csv.py'>, <module 'tablib.formats._tsv' from 'tablib/formats/_tsv.py'>, <module 'tablib.formats._html' from 'tablib/formats/_html.py'>, <module 'tablib.formats._xlsx' from 'tablib/formats/_xlsx.py'>)

Thanks.

XMPP client for monitoring messages from im.kayac.com

2012-01-06T23:17:00.000+09:00

I'm very sorry and somewhat surprised to hear that Notifo will be closed. But there is im.kayac.com.

Notifo provided Android app, but im.kayac.com doesn't. All messages from (to?) im.kayac.com can be forwarded to Google Talk (we need to enroll "api@im.kayac.com" in known list of Google Talk in advance). And Android devices have Google Takl client. Additionally, we can use multiple accounts of Google Talk with imo for Android and imo for iPhone. So we can use im.kayac.com as push notification service corresponding to multiple platform.

We can do a thing that we can't with Notifo. That is monitoring messages from im.kayac.com on PC.

I wrote a XMPP client in Python and tried to monitor messages from im.kayac.com. I used SleekXMPP as library for processing XMPP protocol.

A code for posting a message to im.kayac.com is very simple.

$ python post_im_kayac_com.py 
{"error":"","id":null,"result":"posted"}

$ python bot_im_kayac_com.py 
WARNING  dnspython is not installed -- relying on OS A record resolution
INFO     Negotiating TLS
INFO     Using SSL version: 3
INFO     Node set to: MY_GTAKL_ID/*********

This is a test message

For example, PC can automatically do something and can make Arduino do something in response to messages from im.kayac.com with this XMPP client.

Thanks.

Chrome Extension: Amazon to Yahoo! Bookstore

2011-12-22T22:07:00.000+09:00

Yahoo! Bookstore (Y!B) is a service of Yahoo! Japan. It is similar to iBookstore. Because Y!B sells EPUB files which contain DRM and provides viewer applications for Android and Windows PC (additionally, it shows that it has a plan for providing applications for iOS and Mac).

Incidentally, Apple doesn't launch iBookstore in earnest in Japan. We can download free EPUB files only.

Y!B is new and the functions are inconvenient. We know convenient recommendation system such as Amazon, but Y!B does't have the function.

I forgot to write the other thing. Amazon also doesn't launch Kindle eco-system in Japan. There was report that Amazon will start to sell Kindle and Kindle e-books before the end of the year in Japan. But there isn't follow-up. What's in japanese publishers' mind?

So, I developed a chrome extension for connecting Y!B with Amazon.co.jp.

Chrome Web Store - Amazon to Yahoo! Bookstore

This extension adds buttons for Y!B search link into Amazon's page. By using this, Amazon's great system for discoverying real books can be diverted for discoverying e-books in Y!B.

EPUB files of Y!B are few as compared to iBookstore or Kindle store. But there are many comics comparatively. I hope that major e-books services are launched in Japan as quickly as possible.

Calculation on terminal

2011-12-06T22:29:00.001+09:00

I calculated 1 + 1 in several ways on a terminal when I was free (but I pretended to work).

Rule:

I started with a situation where the terminal shows only $ .
I counted the number of all characters input (including LF) to get answer: 2 and be displayed $ again.



interpreter: command line argument


$ emacs -q -batch -eval '(print (+ 1 1))'  # chars: 40
Loading subst-ksc...
Loading subst-gb2312...
Loading subst-big5...
Loading subst-jis...

2 

$ awk 'BEGIN{print 1+1}'  # chars: 23
2

$ python -c 'print 1+1'   # chars: 22
2

$ perl -e 'print 1+1'     # chars: 20
2 

$ php -r 'echo 1+1;'      # chars: 19
2

$ ruby -e 'p 1+1'         # chars: 16
2


calculator


$ dc -e '1 1+p'  # chars: 14
2

$ echo 1+1|bc    # chars: 12
2

$ expr 1 + 1     # chars: 11
2


bash


$ echo $((1+1))  # chars: 14
2

$ $((1+1))       # chars: 9
bash: 2: command not found 


To tell the truth, my screen's window <0> is always runs Python interpreter. So I don't need to wonder what to use. This is just for the fun of it.



Thanks.



Build tools for Java project
2011-11-26T13:52:00.001+09:00
Apache Maven is hard to understand. I waste time on maintaining pom.xml which lengthens out. Apache Ant's build.xml is in the same situation.



java - Why does Maven have such a bad rep? - Stack Overflow


I looked at List of build automation software and found some good build tools as a substitute for Maven and Ant. There are two requirements for build tools: not writing XML files and resolving dependencies automatically.



SBT and leiningen


SBT is a build tool for Scala project. leiningen is for Clojure. They support Java projects too (but it looks that the support is as a bonus).



We don't need to write XML files for managing project. They requires configuration file written in Scala or Clojure. They use Maven repositories for resolving dependencies.



Buildr


Apache Buildr is based on Ruby. Buildr supports Java, Scala, Groovy and Ruby projects.



Of cource, the configuration files is written Ruby (Rake). It uses Maven repositories for resolving dependencies.



Gradle


Gradle is similar to Buildr and is based on Groovy. Gradle supports  Java, Scala, Groovy and C++ (experimental) projects.



Because Gradle is a follower project, it skims off other build tools. It is raw but is easy to install and use. Gradle's configuration files is written Groovy and it uses Maven repositories for resolving dependencies.



The choice between Buildr and Gradle is the same as choice between Ruby and Groovy. If you be used to Ruby, you should use Buildr. If not, you might like to choice Gradle and write Groovy.



I will rewrite some Maven projects as Gradle projects.



Thanks.






py_compile in Jython
2011-11-12T15:27:00.001+09:00


There is not jythonc in Jython 2.5.x. But there are two modules py_compile and compileall ported from Python.







$ jython -m py_compile Calc.py
$ ls
Calc$py.class Calc.py
$ java -cp /opt/homebrew/Cellar/jython/2.5.2/libexec/jython.jar:. Calc\$py
init Calc
3
-1
6
3


I want to use this class file in Java. But I found that it was a pain.



$ javap Calc\$py
Compiled from ".Users.yoshiyasu.Calc.py"
public class Calc$py extends org.python.core.PyFunctionTable implements org.python.core.PyRunnable{
    static Calc$py self;
    static final org.python.core.PyString _2;
    static final org.python.core.PyInteger _6;
    static final org.python.core.PyInteger _5;
    static final org.python.core.PyString _0;
    static final org.python.core.PyInteger _4;
    static final org.python.core.PyString _1;
    static final org.python.core.PyInteger _3;
    static final org.python.core.PyCode f$0;
    static final org.python.core.PyCode Calc$1;
    static final org.python.core.PyCode __init__$2;
    static final org.python.core.PyCode add$3;
    static final org.python.core.PyCode sub$4;
    static final org.python.core.PyCode mul$5;
    static final org.python.core.PyCode div$6;
    public org.python.core.PyObject f$0(org.python.core.PyFrame, org.python.core.ThreadState);
    public org.python.core.PyObject Calc$1(org.python.core.PyFrame, org.python.core.ThreadState);
    public org.python.core.PyObject __init__$2(org.python.core.PyFrame, org.python.core.ThreadState);
    public org.python.core.PyObject add$3(org.python.core.PyFrame, org.python.core.ThreadState);
    public org.python.core.PyObject sub$4(org.python.core.PyFrame, org.python.core.ThreadState);
    public org.python.core.PyObject mul$5(org.python.core.PyFrame, org.python.core.ThreadState);
    public org.python.core.PyObject div$6(org.python.core.PyFrame, org.python.core.ThreadState);
    public Calc$py(java.lang.String);
    public org.python.core.PyCode getMain();
    public static void main(java.lang.String[]);
    public static org.python.core.CodeBootstrap getCodeBootstrap();
    public org.python.core.PyObject call_function(int, org.python.core.PyFrame, org.python.core.ThreadState);
}


These names of methods are strange and these arguments are org.python.core.PyFrame and org.python.core.ThreadState. When using jythonc, we could specify the methods' names and arguments with @sig docstrings. But py_compile can't read them.



I want to minimize the writing Java code. It is good to use Java from Jython. I will write Scala code when using external codes from Java.



Thanks.






Google Developer Day 2011 Tokyo
2011-11-05T12:42:00.000+09:00


On November 3, Google Developer Day 2011 Japan was held in Pacifico Yokohama. I could participate it because I'm a chrome extension developer and passed DevQuiz.





Because I'm interested in Android, I chose Android sessions of the lot. Android sessions contained the presentations about Android 4.0 (Ice Cream Sandwich), Android UI, ADK, etc. Others, there were sessions about HTML5, Google+ and so on.







Google TV, Chromebook and ADK applications were exhibited. Also, Galaxy Nexus was shown by docomo.





I got Zeemote JS1 that is bluetooth Mobile Gaming Controller for Android. This becomes a bluetooth (HID) mouse for Mac.





I've learned by the great sessions and applications. I want to use the knowledge and skills by creating applications and participate Google Developer Day 2012.



Thanks.






Inserting package statement with vim and emacs
2011-10-29T12:42:00.000+09:00


I didn't previously have much of a sense of package statement of Java because the libraries or applications I made were for only myself. But recently, I'm often well aware of the structure of package on business because I'm in on multiple projects.



When I create a new directory for a project, I use Maven and edit source code files with vim or emacs. If I use a rich IDE (e.g. Eclipse or NetBeans), package statement is inserted automatically into a new file. But I don't use IDE on my office's low-powered computer and it's a pain to check the structure of directories and write package statement by hand each time.



So, I write vim script and emacs lisp for inserting proper package statement into Java files.



Maven creates the below directories by mvn archetype:generate. I create a new Java file in foo directory.



project_dir/
  |_ pom.xml
  |_ src/
       |_ main/
            |_ java/
                 |_ com/
                      |_ foo/
       |_ test/
            |_ java/
                 |_ com/
                      |_ foo/


vim script






# open a new Java file
:call InsertPackagepath()


emacs lisp






# open a new Java file
M-x eval-expression
(insert-packagepath)


Vim script is similar to Python code and easy to be written.



Thanks.






Tools for processing XML
2011-10-22T18:09:00.001+09:00


Apache Oozie is a management system for Apache Hadoop applications. Oozie reads configuration and controls actions described in  XML file (workflow.xml).



It is painful to write XML file directly. I found some tools for helping to write workflow.xml.





xmllint


xmllint is a tool for formatting and validating XML document.



xmllint --format XML_FILE  outputs XML document which is reformatted and reindented.



xmllint --xinclude XML_FILE  outputs XML document which is processed XInclude. For example, a following XML document A is converted to XML document B.







$ xmllint --xinclude --format workflow_a.xml






This document including XInclude element can't be dealt with xmllint (using libxml version 20703) on Mac OS X Lion, but is OK with xmllint (using libxml version 20626) on Linux.



$ xmllint --xinclude workflow_a.xml 
workflow_a.xml:14: element include: XInclude error : XPointer evaluation failed: #xmlns(o=uri:oozie:workflow:0.1)  
    xpointer(//o:action[@name='common_properties']//o:configuration/*)
workflow_a.xml:14: element include: XInclude error : could not load common_properties.xml, and no fallback was found




xsltproc


xsltproc is an engine for processing XSLT. XSLT is a language for transforming XML documents into other XML documents.



Oozie workflow unsupports cyclic flow. So, we need to transform looping workflow into linear workflow. XSLT supports such transformation.





Scala


Scala has scala.xml for handling XML documents as objects.



XSLT is a bit complex and has some  a limit. For example, it is hard to write for-loop in XSLT. We can transform XML document or management multiple XML documents by writing Scala scripts easily.



Oozie is convenient but is unfriendly, I hope that more good tools for Oozie appear.



Thanks.






PerlTeX: Perl + LaTeX
2011-10-08T17:45:00.000+09:00


LuaTeX is Lua + pdfTeX. The documents for LuaTeX can have macro written in Lua.  However, LuaTeX can't use latex document classes. I use pTeX (pLaTeX) for publishing Japanese documents and use jsclasses as document class. But LuaTeX can't use this document class.



I came across PerlTeX in TeXLive. We can write LaTeX macro in Perl with PerlTeX. PerlTeX works as preprocessor, so PerlTeX interprets macro written in Perl and applies the macro to the document. Any LaTeX interpreters can be specified for interpreting the document after PerlTeX processing (default: latex). Therefore, pLaTeX and jsclasses can be used with PerlTeX. PerlTeX is easier to use than LuaTeX for Japanese.



The following document is an example document using PerlTeX.







$ perltex document.tex
This is pdfTeX, Version 3.1415926-2.3-1.40.12 (TeX Live 2011)
 restricted \write18 enabled.
entering extended mode
LaTeX2e <2011/06/27>
Babel  and hyphenation patterns for english, dumylang, nohyphenation, ge
rman-x-2011-07-01, ngerman-x-2011-07-01, afrikaans, ancientgreek, ibycus, arabi
c, armenian, basque, bulgarian, catalan, pinyin, coptic, croatian, czech, danis
h, dutch, ukenglish, usenglishmax, esperanto, estonian, ethiopic, farsi, finnis
h, french, galician, german, ngerman, swissgerman, monogreek, greek, hungarian,
 icelandic, assamese, bengali, gujarati, hindi, kannada, malayalam, marathi, or
iya, panjabi, tamil, telugu, indonesian, interlingua, irish, italian, kurmanji,
 lao, latin, latvian, lithuanian, mongolian, mongolianlmc, bokmal, nynorsk, pol
ish, portuguese, romanian, russian, sanskrit, serbian, serbianc, slovak, sloven
ian, spanish, swedish, turkish, turkmen, ukrainian, uppersorbian, welsh, loaded
.
(./document.tex (/usr/local/texlive/2011/texmf-dist/tex/latex/base/article.cls
Document Class: article 2007/10/19 v1.4h Standard LaTeX document class
(/usr/local/texlive/2011/texmf-dist/tex/latex/base/size10.clo))
(/usr/local/texlive/2011/texmf-dist/tex/latex/perltex/perltex.sty)
(./document.pipe) (./document.pipe) (./document.frpl) (./document.pipe)
(./document.pipe) (./document.frpl)
No file document.aux.
(./document.pipe) (./document.pipe) (./document.frpl) (./document.pipe)
(./document.pipe) (./document.frpl) (./document.pipe) (./document.pipe)
(./document.frpl) (./document.pipe) (./document.pipe) (./document.frpl)
(./document.pipe) (./document.pipe) (./document.frpl) [1] (./document.aux) )
Output written on document.dvi (1 page, 496 bytes).
Transcript written on document.log.




document.lgpl is generated by PerlTeX.



$ cat document.lgpl

############################### PERL CODE ################################
sub latex_displayX {'$x$ is ' . $x}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% LATEX RESULT %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%


############################### PERL CODE ################################
sub latex_reversewords {join " ", reverse split " ", $_[0]}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% LATEX RESULT %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%


############################### PERL CODE ################################
$x = 10
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% LATEX RESULT %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%


############################### PERL CODE ################################
latex_displayX ();

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% LATEX RESULT %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
$x$ is 10

############################### PERL CODE ################################
$x = $x * 10
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% LATEX RESULT %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%


############################### PERL CODE ################################
latex_displayX ();

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% LATEX RESULT %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
$x$ is 100

############################### PERL CODE ################################
latex_reversewords ('The quick brown fox jumps over the lazy dog');

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% LATEX RESULT %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
dog lazy the over jumps fox brown quick The


We can know how macro in the document works by reading document.lgpl.



--latex option is used for specifying LaTeX interpreters.



$ perltex --latex=platex document.tex 
This is e-pTeX, Version 3.1415926-p3.2-110415-2.3 (utf8.euc) (TeX Live 2011)
 restricted \write18 enabled.
entering extended mode
pLaTeX2e <2006/11/10> (based on LaTeX2e <2011/06/27> patch level 0)
...


LaTeX macro is a little bit complicated. PerlTeX helps writing small macro for processing text.



Thanks.











Converting relative link to absolute link
2011-09-24T18:35:00.000+09:00


When parsing HTML files, the relative links need to be converted to absolute links.



In Python, urlparse.urljoin is suitable for converting.



>>> import urlparse

>>> urlparse.urljoin("http://www.example.org/", "index.html")
'http://www.example.org/index.html'  # OK

>>> urlparse.urljoin("http://www.example.org", "another.html")
    # base-url doesn't have path
'http://www.example.org/another.html'  # OK


In Java, java.net.URI.resolve is suitable. Here, I show an example by Jython.



>>> from java.net import URI

>>> URI("http://www.example.org/").resolve("index.html").toString()
u'http://www.example.org/index.html'  # OK

>>> URI("http://www.example.org").resolve("another.html").toString()
    # base-url doesn't have path
u'http://www.example.organother.html'  # BAD!


This behavior is hard to be found out because "http://www.example2.org" is a right URI and the constructor of java.net.URI doesn't throws java.net.URISyntaxException. Additionally, "http://www.example2.organother.html" is a right URI, too.



Therefore, I need to add a code for cheking that the base-url has path or not.





>>> uriNoPath = "http://www.example.org"
>>> example = URI(uriNoPath)
>>> if example.path == '':
...         example = URI(uriNoPath + '/')
... 
>>> example.resolve("index.html")
http://www.example.org/index.html


Thanks.







Checking if a Perl module is installed
2011-09-17T21:22:00.001+09:00


We can run an instant code with one-liner with using -e and -M options. By using this, I can check if a Perl module is installed.



$ perl -MLWP -e ''

$ perl -Mlocal::lib -e ''
Can't locate local/lib.pm in @INC (@INC contains: /Library/Perl/5.12/darwin-thread-multi-2level /Library/Perl/5.12 /Network/Library/Perl/5.12/darwin-thread-multi-2level /Network/Library/Perl/5.12 /Library/Perl/Updates/5.12.3 /System/Library/Perl/5.12/darwin-thread-multi-2level /System/Library/Perl/5.12 /System/Library/Perl/Extras/5.12/darwin-thread-multi-2level /System/Library/Perl/Extras/5.12 .).
BEGIN failed--compilation aborted.


Bash special variable: $? sets the exit status of the last command.



$ ls /Network; echo $?
0

$ ls NOTFOUND; echo $?
ls: NOTFOUND: No such file or directory
1


I wrote a bash function for checking if a Perl module is installed in combination them.







$ CheckPerlModule File::Path
OK

$ CheckPerlModule Jcode
NG


Thanks.







Extracting lines including the specified IP addresses from logs
2011-09-10T13:31:00.000+09:00


I needed to extract lines which include the specified range of IP addresses from httpd log files. The log file was large size and the fast processing was required So, I tried some methods.



Here, I used httpd access logs in my web server. The total size of log files is about 19MB (72733 lines). I extracted lines which include IP addresses of University of Tokyo from the log files.



A range of IP addresses expresses in CIDR. The rangees of IP addresses of University of Tokyo are 130.69.0.0/16, 133.11.0.0/16 and 157.82.0.0/16.



with Perl


There is Net::IP::Match::Regexp for checking that an IP address is contained in a range of IP addresses.



I installed this module from CPAN.









with Python


There are ipcalc and netaddr for checking that an IP address is contained in a range of IP addresses.



I installed these module from PyPI using pip.













with Haskell


There is Data.IP for checking that an IP address is contained in a range of IP addresses.



This module is included in iproute package. I installed this package using cabal.









Comparison


$ wc -l /var/log/lighttpd/access.log*
    4141 /var/log/lighttpd/access.log
    4688 /var/log/lighttpd/access.log.1
    5019 /var/log/lighttpd/access.log.10
    6265 /var/log/lighttpd/access.log.11
    8291 /var/log/lighttpd/access.log.12
    5657 /var/log/lighttpd/access.log.2
    4850 /var/log/lighttpd/access.log.3
    4626 /var/log/lighttpd/access.log.4
    4963 /var/log/lighttpd/access.log.5
    6795 /var/log/lighttpd/access.log.6
    4580 /var/log/lighttpd/access.log.7
    5804 /var/log/lighttpd/access.log.8
    7054 /var/log/lighttpd/access.log.9
   72733 total


$ time cat /var/log/lighttpd/access.log* | perl ipfilter.pl | wc -l
     612

real    0m2.355s
user    0m2.306s
sys     0m0.036s

$ time cat /var/log/lighttpd/access.log* | python ipfilter_ipcalc.py | wc -l
     612

real    0m6.868s
user    0m6.782s
sys     0m0.077s

$ time cat /var/log/lighttpd/access.log* | python ipfilter_netaddr.py | wc -l
     612

real    0m3.932s
user    0m3.861s
sys     0m0.062s

$ time cat /var/log/lighttpd/access.log* | runhaskell ipfilter.hs | wc -l
     612

real    0m3.706s
user    0m3.599s
sys     0m0.101s


ipcalc/Python is the slowest. The reason is that ipcalc.Network.in_network(other) calls long(address) many times.



iproute/Haskell uses TRIE for storing IP addresses. If the code is compiled with GHC, the better result may return.



I was impressed by the result of Net::IP::Match::Regexp/Perl. As may be expected of a standard tool for text processing, Perl is strong in a job like this.



Thanks.







compiling gc-7.1 on Mac OS X Lion, it worked?
2011-09-02T19:52:00.001+09:00


In the previous entry, I failed to compile gc-7.1 on Mac OS X Lion because I used llvm-gcc.



I found there were GNU C Compiler: gcc-4.2 and g++-4.2 on Lion. So, I tried to compile gc-7.1 again. As a result, It was apparently successful, but the test was failed.



I needed to edit source code to avoid compile error. One is to replace <ucontext.h> to <sys/ucontext.h>, the other is to delete INLINE.



$ diff gc-7.1_orig gc-7.1
diff gc-7.1_orig/alloc.c gc-7.1/alloc.c
875c875
< static INLINE word GC_max(word x, word y)
---
> static word GC_max(word x, word y)
880c880
< static INLINE word GC_min(word x, word y)
---
> static word GC_min(word x, word y)
diff gc-7.1_orig/mach_dep.c gc-7.1/mach_dep.c
163c163
< # include <ucontext.h>
---
> # include <sys/ucontext.h>
diff gc-7.1_orig/mark_rts.c gc-7.1/mark_rts.c
100c100
< static INLINE int rt_hash(ptr_t addr)
---
> static int rt_hash(ptr_t addr)
diff gc-7.1_orig/os_dep.c gc-7.1/os_dep.c
2755c2755
< #   include <ucontext.h>
---
> #   include <sys/ucontext.h>



$ CC=gcc-4.2 CXX=g++-4.2 ./configure
$ CC=gcc-4.2 CXX=g++-4.2 make
$ CC=gcc-4.2 CXX=g++-4.2 make check
FAIL: gctest
FAIL: leaktest
PASS: middletest
PASS: smashtest
PASS: hugetest
PASS: threadleaktest



mmm...


Thanks.





Xcode and github
2011-08-27T15:06:00.000+09:00


I found that Xcode 4 had git frontend. I use mercurial for version control of programs or documents because I'm a  Pythonista. I wasn't bothered by them which do I prefer. Thus, I decided to use git for Xcode projects and use mercurial for other projects.



There is bitbucket for mercurial. There is github for git. I tried to correspond with github repositories and Xcode.



github settings


I had to enregister public ssh key for using github. This time I created a dedicated ssh key pair for github.



$ ssh-keygen -t rsa -C "laclef.yoshiyasu@gmail.com"
    # I set name specified as "id_rsa_github"

$ cat >> ~/.ssh/config

Host github.com
    User git 
    Hostname github.com
    PreferredAuthentications publickey
    IdentityFile ~/.ssh/id_rsa_github

(C-d)

$ cat .ssh/id_rsa_github.pub |pbcopy


I pasted the public ssh key to github settings.



For checking setting, I accessed github.com with ssh.



$ ssh -T github.com
The authenticity of host 'github.com (207.97.227.239)' can't be established.
RSA key fingerprint is 16:27:ac:a5:76:28:2d:36:63:1b:56:4d:eb:df:a6:48.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'github.com,207.97.227.239' (RSA) to the list of known hosts.
Identity added: /Users/yoshiyasu/.ssh/id_rsa_github (/Users/yoshiyasu/.ssh/id_rsa_github)
Hi laclefyoshi! You've successfully authenticated, but GitHub does not provide shell access.


In the first accessing, SecurityAgent opened.





I created a new repository on github.





And I took notes of URI of this repository.





Xcode


For importing github repository to Xcode, I chose a menu item: [File] - [Source Control] - [Repositories...].





"Organizer - Repositories" window opened. I selected "Added Repository..." and inputed URI of the github repository.









When "Clone" button was pushed, the github repository (origin) was cloned as a local reposotory (master).



I created a file: File.c in a local repository and committed it.









I couldn't find a item for pushing the local repository to github repository. So I needed to use Terminal.app and git command.



$ cd Documents/Projects/xcode/First-repository/
$ ls -la
total 8
drwxr-xr-x   4 yoshiyasu  staff   136B  8 27 12:23 ./
drwxr-xr-x  11 yoshiyasu  staff   374B  8 27 12:19 ../
drwxr-xr-x  13 yoshiyasu  staff   442B  8 27 12:26 .git/
-rw-r--r--@  1 yoshiyasu  staff   167B  8 27 12:23 File.c

$ git log
commit a8d7253f3d7bb943659266305aa5e364befe9054
Author: SAEKI Yoshiyasu <laclef.yoshiyasu@gmail.com>
Date:   Sat Aug 27 12:26:44 2011 +0900

    added File.c

$ git push origin master
Counting objects: 3, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (2/2), done.
Writing objects: 100% (3/3), 371 bytes, done.
Total 3 (delta 0), reused 0 (delta 0)
To git@github.com:laclefyoshi/First-repository.git
 * [new branch]      master -> master


I checked the github repository.







Branch


I could create branches of a repository in Xcode.



For creating branches, I pushed "Add Branch" button of the repository's Branches menu on "Organizer - Repositories" window.





I switched (checkout) the created branch and edited and committed a file. The diff of file can be shown on the commit window.





I pushed the local repository to the github repository. So two branches were shown.



$ git branch
* a-branch
  master

$ git push origin a-branch
Counting objects: 5, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (2/2), done.
Writing objects: 100% (3/3), 322 bytes, done.
Total 3 (delta 1), reused 0 (delta 0)
To git@github.com:laclefyoshi/First-repository.git
 * [new branch]      a-branch -> a-branch
 




In Xcode, I couldn't merge the branches and used git command.



$ git checkout master
Switched to branch 'master'

$ git merge a-branch
Updating a8d7253..3a1669d
Fast-forward
 File.c |    1 +
 1 files changed, 1 insertions(+), 0 deletions(-)


Apparently, the items of [Source Contorol] - [Commit], [Push], [Pull], [Update], [Merge] and so on become available when creating and opening  Xcode project. When only cloning a git repository, these items couldn't be used.



Thanks.







compiling gc-7.1 on Mac OS X Lion, but error
2011-08-24T22:19:00.001+09:00
For compiling gc-7.1 (boehm-gc) on Mac OS X Lion (Version 10.7) and Xcode 4.1, I had to edit some source files. gcc on Lion is not GNU Compiler Collection, it is llvm.



$ gcc -v
Using built-in specs.
Target: i686-apple-darwin11
Configured with: /private/var/tmp/llvmgcc42/llvmgcc42-2335.15~25/src/configure --disable-checking --enable-werror --prefix=/Developer/usr/llvm-gcc-4.2 --mandir=/share/man --enable-languages=c,objc,c++,obj-c++ --program-prefix=llvm- --program-transform-name=/^[cg][^.-]*$/s/$/-4.2/ --with-slibdir=/usr/lib --build=i686-apple-darwin11 --enable-llvm=/private/var/tmp/llvmgcc42/llvmgcc42-2335.15~25/dst-llvmCore/Developer/usr/local --program-prefix=i686-apple-darwin11- --host=x86_64-apple-darwin11 --target=i686-apple-darwin11 --with-gxx-include-dir=/usr/include/c++/4.2.1
Thread model: posix
gcc version 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2335.15.00)


$ ./configure
$ make
...
 gcc -DPACKAGE_NAME=\"gc\" -DPACKAGE_TARNAME=\"gc\" -DPACKAGE_VERSION=\"7.1\" "-DPACKAGE_STRING=\"gc 7.1\"" -DPACKAGE_BUGREPORT=\"Hans.Boehm@hp.com\" -DGC_VERSION_MAJOR=7 -DGC_VERSION_MINOR=1 -DPACKAGE=\"gc\" -DVERSION=\"7.1\" -DGC_DARWIN_THREADS=1 -DTHREAD_LOCAL_ALLOC=1 -DHAS_X86_THREAD_STATE32___EAX=1 -DSTDC_HEADERS=1 -DHAVE_SYS_TYPES_H=1 -DHAVE_SYS_STAT_H=1 -DHAVE_STDLIB_H=1 -DHAVE_STRING_H=1 -DHAVE_MEMORY_H=1 -DHAVE_STRINGS_H=1 -DHAVE_INTTYPES_H=1 -DHAVE_STDINT_H=1 -DHAVE_UNISTD_H=1 -DHAVE_DLFCN_H=1 -DNO_EXECUTE_PERMISSION=1 -DALL_INTERIOR_POINTERS=1 -DGC_GCJ_SUPPORT=1 -DJAVA_FINALIZATION=1 -DATOMIC_UNCOLLECTABLE=1 -I./include -fexceptions -I libatomic_ops/src -g -O2 -MT os_dep.lo -MD -MP -MF .deps/os_dep.Tpo -c os_dep.c  -fno-common -DPIC -o .libs/os_dep.o
libatomic_ops/src/atomic_ops/sysdeps/gcc/x86_64.h: In function 'AO_test_and_set_full':
libatomic_ops/src/atomic_ops/sysdeps/gcc/x86_64.h:127: error: unsupported inline asm: input constraint with a matching output constraint of incompatible type!
make[1]: *** [os_dep.lo] Error 1
make: *** [all-recursive] Error 1


In libatomic_ops/src/atomic_ops/sysdeps/gcc/x86_64.h, there is a function: AO_test_and_set_full. This function uses inline assembler. I had to make the code responsive to llvm-gcc.



$ diff libatomic_ops/src/atomic_ops/sysdeps/gcc/x86_64.h.org libatomic_ops/src/atomic_ops/sysdeps/gcc/x86_64.h
129c129
<               : "0"(0xff), "m"(*addr) : "memory");
---
>               : "0"((unsigned char)0xff), "m"(*addr) : "memory");


$ make
...
 gcc -DPACKAGE_NAME=\"gc\" -DPACKAGE_TARNAME=\"gc\" -DPACKAGE_VERSION=\"7.1\" "-DPACKAGE_STRING=\"gc 7.1\"" -DPACKAGE_BUGREPORT=\"Hans.Boehm@hp.com\" -DGC_VERSION_MAJOR=7 -DGC_VERSION_MINOR=1 -DPACKAGE=\"gc\" -DVERSION=\"7.1\" -DGC_DARWIN_THREADS=1 -DTHREAD_LOCAL_ALLOC=1 -DHAS_X86_THREAD_STATE32___EAX=1 -DSTDC_HEADERS=1 -DHAVE_SYS_TYPES_H=1 -DHAVE_SYS_STAT_H=1 -DHAVE_STDLIB_H=1 -DHAVE_STRING_H=1 -DHAVE_MEMORY_H=1 -DHAVE_STRINGS_H=1 -DHAVE_INTTYPES_H=1 -DHAVE_STDINT_H=1 -DHAVE_UNISTD_H=1 -DHAVE_DLFCN_H=1 -DNO_EXECUTE_PERMISSION=1 -DALL_INTERIOR_POINTERS=1 -DGC_GCJ_SUPPORT=1 -DJAVA_FINALIZATION=1 -DATOMIC_UNCOLLECTABLE=1 -I./include -fexceptions -I libatomic_ops/src -g -O2 -MT mach_dep.lo -MD -MP -MF .deps/mach_dep.Tpo -c mach_dep.c  -fno-common -DPIC -o .libs/mach_dep.o
In file included from mach_dep.c:163:
/usr/include/ucontext.h:43:2: error: #error The deprecated ucontext routines require _XOPEN_SOURCE to be defined
make[1]: *** [mach_dep.lo] Error 1
make: *** [all-recursive] Error 1


There is not /usr/include/ucontext.h on Mac. But there is /usr/include/sys/ucontext.h.



$ ack ucontext.h *
mach_dep.c
163:# include <ucontext.h>

os_dep.c
2755:#   include <ucontext.h>




$ diff mach_dep.c.org mach_dep.c
163c163
< # include <ucontext.h>
---
> # include <sys/ucontext.h>

$ diff os_dep.c.org os_dep.c
2755c2755
< #   include <ucontext.h>
---
> #   include <sys/ucontext.h>


$ make
...
(cd .libs && rm -f libcord.1.dylib && ln -s libcord.1.0.3.dylib libcord.1.dylib)
(cd .libs && rm -f libcord.dylib && ln -s libcord.1.0.3.dylib libcord.dylib)
ar cru .libs/libcord.a  cord/cordbscs.o cord/cordprnt.o cord/cordtest.o cord/cordxtra.o
ranlib .libs/libcord.a
creating libcord.la
(cd .libs && rm -f libcord.la && ln -s ../libcord.la libcord.la)
$ 


Done? I checked.





$ make check
...
tests/thread_leak_test.c: In function ‘main’:
tests/thread_leak_test.c:38: warning: format ‘%lu’ expects type ‘long unsigned int’, but argument 2 has type ‘int’
tests/thread_leak_test.c:38: warning: format ‘%lu’ expects type ‘long unsigned int’, but argument 2 has type ‘int’
/bin/sh ./libtool --tag=CC --mode=link gcc -fexceptions -I libatomic_ops/src -g -O2   -o threadleaktest  tests/thread_leak_test.o ./libgc.la  
gcc -fexceptions -I libatomic_ops/src -g -O2 -o .libs/threadleaktest tests/thread_leak_test.o  ./.libs/libgc.dylib -lpthread
creating threadleaktest
make  check-TESTS
Switched to incremental mode
Emulating dirty bits with mprotect/signals
List reversal produced incorrect list - collector is broken
Test failed
/bin/sh: line 1: 55042 Abort trap: 6           ${dir}$tst
FAIL: gctest
Leaked composite object at 0x10162cee0 (tests/leak_test.c:12, sz=13, NORMAL)

Leaked composite object at 0x10162cf20 (tests/leak_test.c:12, sz=12, NORMAL)

Leaked composite object at 0x10162cf60 (tests/leak_test.c:12, sz=11, NORMAL)

Leaked composite object at 0x10162cfa0 (tests/leak_test.c:12, sz=10, NORMAL)

Leaked composite object at 0x10162cfe0 (tests/leak_test.c:12, sz=9, NORMAL)

Leaked composite object at 0x10162bf20 (tests/leak_test.c:12, sz=8, NORMAL)

Leaked composite object at 0x10162bf50 (tests/leak_test.c:12, sz=7, NORMAL)

Leaked composite object at 0x10162bf80 (tests/leak_test.c:12, sz=6, NORMAL)

Leaked composite object at 0x10162bfb0 (tests/leak_test.c:12, sz=5, NORMAL)

Leaked composite object at 0x10162bfe0 (tests/leak_test.c:12, sz=4, NORMAL)

GC_debug_free: found smashed location at 0x10162bfa8 in or near object at 0x10162bfb0(0�b:0, sz=0)
GC_debug_free: found smashed location at 0x10162bf78 in or near object at 0x10162bf80(EMPTY(smashed?):0, sz=0)
GC_debug_free: found smashed location at 0x10162bf48 in or near object at 0x10162bf50(EMPTY(smashed?):0, sz=0)
GC_debug_free: found smashed location at 0x10162bf18 in or near object at 0x10162bf20(, appr. sz = 9)
GC_debug_free: found smashed location at 0x10162cfd8 in or near object at 0x10162cfe0(@�b:0, sz=0)
GC_debug_free: found smashed location at 0x10162cf98 in or near object at 0x10162cfa0(EMPTY(smashed?):0, sz=0)
GC_debug_free: found smashed location at 0x10162cf58 in or near object at 0x10162cf60(��b:0, sz=0)
GC_debug_free: found smashed location at 0x10162cf18 in or near object at 0x10162cf20(EMPTY(smashed?):0, sz=0)
GC_debug_free: found smashed location at 0x10162ced8 in or near object at 0x10162cee0(, appr. sz = 25)
/bin/sh: line 1: 55061 Segmentation fault: 11  ${dir}$tst
FAIL: leaktest
Final heap size is 131072
PASS: middletest
GC_check_heap_block: found smashed heap objects:
0x10df1dfe8 in or near object at 0x10df1dfc0(tests/smash_test.c:21, sz=40)
GC_check_heap_block: found smashed heap objects:
0x10df1dfe8 in or near object at 0x10df1dfc0(tests/smash_test.c:21, sz=40)
GC_check_heap_block: found smashed heap objects:
0x10df90f98 in or near object at 0x10df90f70(tests/smash_test.c:21, sz=40)
0x10df1dfe8 in or near object at 0x10df1dfc0(tests/smash_test.c:21, sz=40)
PASS: smashtest
GC Warning: Out of Memory!  Returning NIL!
GC Warning: Out of Memory!  Returning NIL!
GC Warning: Out of Memory!  Returning NIL!
PASS: hugetest
Leaked composite object at 0x10b7a8fe0 (tests/thread_leak_test.c:14, sz=4, NORMAL)

Leaked composite object at 0x10b7a8f80 (tests/thread_leak_test.c:14, sz=4, NORMAL)

Leaked composite object at 0x10b7a8ef0 (tests/thread_leak_test.c:14, sz=4, NORMAL)

Leaked composite object at 0x10b7a8d40 (tests/thread_leak_test.c:14, sz=4, NORMAL)

Leaked composite object at 0x10b7a8e30 (tests/thread_leak_test.c:14, sz=4, NORMAL)

PASS: threadleaktest
==================================
2 of 6 tests failed
Please report to Hans.Boehm@hp.com
==================================
make[2]: *** [check-TESTS] Error 1
make[1]: *** [check-am] Error 2
make: *** [check-recursive] Error 1


It failed in gctest and leaktest and Segmentation fault occurred.



I went under and found the github repository on Wikipedia: Boehm garbage collector. It would appear that the repository is updated frequently. I think there is hope.



Thanks.







Hello, Lion!
2011-08-21T21:20:00.001+09:00
I bought Mac mini (Mid 2011) on this Saturday.



This default OS is Mac OS X Lion 10.7. Because Lion is is  different from Snow Leopard very well, it took me a while to get used to this change.



Firstly, gcc is llvm-gcc. This impact has a tremendous effect for me. For example, gc-7.1 is required to modify the source to be compiled.



$ ls -l `which gcc`
lrwxr-xr-x  1 root  wheel    12B  8 21 00:50 /usr/bin/gcc@ -> llvm-gcc-4.2


Secondly, there is not Rosetta in Lion. So, some PPC applications are not working. Therefore I had to seek the response. GanttProject for Lion is here. Jabref became Lion-compliant in ver.2.7.





I also feel in sense of embarrassment by Mission Control. I don't receive the benefits of Auto save or Versions yet. From now, I experiment on Lion's new feature.



Thanks.







Basic network analysis and visualization with igraph
2011-08-13T14:58:00.000+09:00


igraph is a library for analysis and visualization of network graph. I have gotten more chances to analyze network graph recently.



Using igraph in Python


$ mkvirtualenv igraph
(igraph)$ ./configure --prefix=$HOME/.virtualenvs/igraph
(igraph)$ make
(igraph)$ make install
(igraph)$ export C_INCLUDE_PATH=$HOME/.virtualenvs/igraph/include:$C_INCLUDE_PATH
(igraph)$ export LIBRARY_PATH=$HOME/.virtualenvs/igraph/lib:$LD_LIBRARY_PATH
(igraph)$ pip install python-igraph


(igraph)$ igraph
No configuration file, using defaults
igraph 0.5.4 running inside Python 2.7.2 (default, Jul 15 2011, 23:46:46) 
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
# == from igraph import * (in Python)
>>> g = Graph.Barabasi(30)
>>> print g
Undirected graph (|V| = 30, |E| = 29)
>>> print g.vcount(), g.ecount()
30 29
>>> [m for m in dir(g) if m.startswith("write")]
['write', 'write_adjacency', 'write_dimacs', 'write_dot', 'write_edgelist', 'write_gml', 'write_graphml', 'write_graphmlz', 'write_lgl', 'write_ncol', 'write_pajek', 'write_pickle', 'write_svg']
>>> [m for m in dir(g) if m.startswith("layout")]
['layout', 'layout_circle', 'layout_drl', 'layout_fruchterman_reingold', 'layout_fruchterman_reingold_3d', 'layout_graphopt', 'layout_grid_fruchterman_reingold', 'layout_kamada_kawai', 'layout_kamada_kawai_3d', 'layout_lgl', 'layout_random', 'layout_random_3d', 'layout_reingold_tilford', 'layout_reingold_tilford_circular', 'layout_sphere', 'layout_star']
>>> g.write_svg("graph.svg", layout=g.layout_kamada_kawai())




>>> g.vs[0]["color"] = "red"
>>> g.vs[0]        
igraph.Vertex(,0,{'color': 'red'})
>>> g.es[1].tuple
(0, 2)
>>> g.write_svg("graph2.svg", layout=g.layout_kamada_kawai())




igraph can calculate PageRank of directed graph.



>>> g1 = Graph(4, directed=True)
>>> g1.add_edges([(0,1),(0,2),(1,0),(1,2),(3,0)])
>>> g1.edge_betweenness()
[2.0, 2.0, 1.0, 1.0, 3.0]
>>> g1.degree()
[4, 3, 2, 1]
>>> [v.index for v in g1.vs.select(_degree = g1.maxdegree())]
[0]
>>> print g1.summary()
4 nodes, 5 edges, directed

Number of components: 3
Diameter: 2
Density: 0.4167
Reciprocity: 0.2500
Average path length: 1.2857
>>> g1.average_path_length()        
1.2857142857142858
>>> g1.pagerank()
[0.2910808248180501, 0.24604410843705093, 0.40564107854853176, 0.05723398819636713]
>>> g.write_svg("pagegraph.svg", layout=g.layout_kamada_kawai())




Using igraph in R


> install.packages("igraph")
> library("igraph")


igraph can read external file as edges information.



> system("cat edges.txt")
0 1
0 2
1 0
1 2
3 0
> g <- graph.data.frame(read.delim("edges.txt", sep=" ", h=FALSE, as.is=T), directed=TRUE)
    # == graph.edgelist(as.matrix(read.delim("edges.txt", sep=" ", h=FALSE, as.is=T)), directed=TRUE)
> g
Vertices: 4 
Edges: 5 
Directed: TRUE 
Edges:
              
[0] '0' -> '1'
[1] '0' -> '2'
[2] '1' -> '0'
[3] '1' -> '2'
[4] '3' -> '0'

> V(g)
Vertex sequence:
[1] "0" "1" "3" "2"
> E(g)
Edge sequence:
          
[0] 0 -> 1
[1] 0 -> 2
[2] 1 -> 0
[3] 1 -> 2
[4] 3 -> 0
> average.path.length(g)
[1] 1.285714
> degree(g)
[1] 4 3 1 2
> edge.betweenness(g)
[1] 2 2 1 1 3
> page.rank(g)$vector
[1] 0.29108082 0.24604411 0.05723399 0.40564108
> plot(g, layout=layout.kamada.kawai)



igraph can handle adjacency matrix.

> graph.adjacency(matrix(c(0,1,1,0,1,0,1,0,0,0,0,0,1,0,0,0), nrow=4, ncol = 4, byrow = TRUE))
Vertices: 4 
Edges: 5 
Directed: TRUE 
Edges:
          
[0] 0 -> 1
[1] 0 -> 2
[2] 1 -> 0
[3] 1 -> 2
[4] 3 -> 0


igraph for Python and R have much the same API. Therefore, I can use igraph easily in every aspect.



Thanks.







schema.org for people or machines
2011-08-06T21:42:00.001+09:00
schema.org is a collection of terms for describing contents of web page. Google, Yahoo! and Bing think that the search result view is better with reader these embedded terms in web pages. The terms of schema.org is one of metadata terms. And already, schema.rdfs.org provides RDF Schema documents for schema.org. 



But there is no terms for describing metadata of links. For example, the terms don't include "nofollow". Additionary there are not "noindex" or "noarchive". Thus, schema.org is not for controlling search engine (robot). In my mind, links of web pages have meanings. If these meaning are described, better search result can be provided.



Truthfully, these meaning should be read by AI-programmed robot. But it is difficult that the current AI correctly reads semistructured data like HTML documents. I guess that Google is going to gather the metadata for learning of AI.



People with machines will create the better environment for information system.



Happy 20th Birthday, World Wide Web!



Thanks.





DLNA in my room
2011-07-30T20:17:00.000+09:00
I bought a TV: SONY BRAVIA KDL-32EX420. In Japan, NTSC was stopped on July 24 and DTTV has launched. A TV which I had can't screen DTTV, so I needed to replace it.



The BRAVIA has DLNA client feature. I tried to use it with Android phone and Macbook.



Firstly, I installed DLNA server: TVMOBiLi on my Macbook. I needed to select "DEVELOPMENT VERSION" (2.0.0.3163/Jul 13 2011) for communicating with BRAVIA. Secondly, I installed Twonky Mobile on my Android phone. Twonky Mobile becomes DLNA server and client.



Of course, they must be located at the same network for linking together with DLNA.



BRAVIA may not be able to find DLNA servers just after starting the DLNA servers. I needed to wait for about five minutes. When the DLNA servers are displayed on media menu of BRAVIA, it's successful. I can select and play MP3, MP4 and  JPG files.







Additionally, Twonky can transfer playable media to DLNA client. This feature is called "Beam".





Remote control of BRAVIA doesn't use infrared ray and use RF (RF4CE). Furthermore, Sony provides a nice remote controller application for Android and iOS. This application uses Wifi. 





In my room, most of devices can communicate with each other through LAN. Some devices are working as server, others are working as client. An Android phone in my hand controls these devices directly or indirectly. This convenient environment is  very similar to a sight which I imagined when I was a child.  The advancement of technology is great.



Thanks.





Protocol Buffers Inheritance
2011-07-23T12:46:00.000+09:00
For describing inheritance of messages with Protocol Buffers, there are some techniques. I thought about each merits and demerits since the previous entry.



I decided to use intension with extensions/extend by reference to RMS and Protocol Buffers « Technofinancial Singularity, Protocol Buffer Polymorphism and Bringing EMF Ecore and Google Protocol Buffers together « Christian Kerl.















$ read_person
Person ID: 0
        Name: you
Person ID: 1
        Name: saeki
        He is flying!


By using extensions/extend for  describing inheritance, I don't need to rewrite a parent message every time a child message is added. Furthermore, a child message can be assigned to a field which is designated as the parent message.



Thanks.





Protocol Buffers
2011-07-16T14:19:00.001+09:00
There are many serialization formats. For example, there are pickle for Python, Serializable API for Java, and so on. These choices are dependent on the used programming language.



On the other hand, there are some serialization formats which are independence on the used programming language. For exmaple, there are XML, JSON, Protocol Buffers, and so on.



I started to use Protocol Buffers because I need to write a schema of the data for serialization and require the faster behavior.



I wrote a simple proto code and C++ code after reading tutorial.







$ protoc person.proto --cpp_out=dist










$ g++ -o add_person add_person.cc person.pb.cc `pkg-config --cflags --libs protobuf`
$ ./add_person

$ cat -v PersonBook  # for showing nonprinting characters
^H^A^R^Esaeki^Z^Qsaeki@example.org^H^B^R^Hfurukawa^Z^Tfurukawa@example.org

$ g++ -o read_person read_person.cc person.pb.cc `pkg-config --cflags --libs protobuf`
$ ./read_person
Person ID: 2
        Name: furukawa
        Email: furukawa@example.org


I didn't set a manager class for Person. So this read_person can read an only last Person.



I wrote a Python code, too.



$ protoc person.proto --python_out=dist






$ python  read_person.py 
Person ID: 2
        Name: furukawa
        Email: furukawa@example.org






I want to represent classes inheritance. But Protocol Buffers doesn't provide the way for representing them. But there a way as to use intension; ref.Protocol Buffer Polymorphism.



Isn't there any good way?



Thanks.