Software for humans

My experience with Btrfs on Debian - no space on device

2015-03-24T22:32:00.003+02:00

I wish to share with you some of my experiances with BTRFS. Maybe it will be usefull.

I'm using btrfs on my /home partition, on my Debian laptop for a few months now. All seemed to be pretty good. The only issues that I had was the fact that filesystem access seemed to be a bit slow on some occasions, but generally things where ok.

This went on until today, when I noticed a complete slowdown of the system, to a point that it became unusable. Some symptoms: Google Chrome crashed and became unusable, removing files was very slow (rm -rf node_modules, nautilus deletion, etc. ) Intellij WebStorm was barelly starting, node.js was complaining about 'not enough space on device' .

I also checked my root partition to validate if the prbolem was btrfs related or not. My root partition, which is ext4 and I created/removed some files. Operations performed normally.

My system was thus unusable and I didn't had any idea why.

So I started investigating...

I checked btrfs file system size, but things where ok:

btrfs fi df /home showed about 10gb of free space

sudo btrfs fi df /home/
Data, single: total=229.80GiB, used=218.80GiB
System, single: total=32.00MiB, used=20.00KiB
Metadata, single: total=3.00GiB, used=2.49GiB
GlobalReserve, single: total=512.00MiB, used=0.00B

My fstab looked like this (I had cow active):

/dev/sda3 /home btrfs autodefrag,compress=lzo,space_cache 0 0

Not knowing what to do, I started changing /etc/fstab and rebooting my laptop. First I activated noatime, then I deactivated cow (with nodataow) but I did not notice any improvement.

The solution:

I was very close to re-install Debian on my machine but decided to give it one more chance. I didn't like that thought, since it would have ment a lot of work. I seated myself at the keyboard and started a btrfs defrag

"btrfs filesystem defrag /home"

While I was at it, I thought it will be nice to have some more free-space so I started to delete a bunch of big files. That took a few minutes and I got some extra space and Evrika !!! my system was usable once more !

Data, single: total=229.80GiB, used=199.81GiB
System, single: total=32.00MiB, used=20.00KiB
Metadata, single: total=3.00GiB, used=2.49GiB
GlobalReserve, single: total=512.00MiB, used=0.00B

My current /etc/fstab :

/dev/sda3 /home btrfs nodatacow,autodefrag,noatime,compress=lzo,space_cache 0 0

What caused this?

I'm not very sure. It's kind of obvious that I've hit the btrfs low disk space problem. The strange thing is that I've been having ~ 10 Gb of free space for a long time (more than a 1-2 months) and I did not notice that behaviour. After more thought of what I did different the laste few days is that I started using some node.js build tools that generate a lot of writes on change ( broccoli js build tool, ember-cli ). I will continue to investigate this, but right now I'm happy to have a working system and will also migrate my root partition to btrfs once Debian Jessie is released.

On the down side, it's a pitty that with btrfs you can do so much, but you can't use all your disk space, and you can make your system very unstable very easy.

$ sudo btrfs fi show
Label: none uuid: 49d9c741-b00b-487b-be5c-49da02325a16
Total devices 1 FS bytes used 202.30GiB
devid 1 size 232.83GiB used 232.83GiB path /dev/sda3

Btrfs v3.17

$ uname -a

Linux daos 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt7-1 (2015-03-01) x86_64 GNU/Linux

Update:
More 'scientific' artiles related to the subject:
http://unix.stackexchange.com/questions/174446/btrfs-error-error-during-balancing-no-space-left-on-device
http://marc.merlins.org/perso/btrfs/post_2014-05-04_Fixing-Btrfs-Filesystem-Full-Problems.html

Modular REST applications with Karaf features for OSGi Jax-RS Connector

2014-10-17T00:03:00.002+03:00

The purpose of this article is to let you know how easy it is to develop modular REST (JAX-RS) applications on OSGi, particularly Apache Karaf.

For some time I'm working on improving the way I deliver applications. My focus is on quality, ease of understanding and speed of delivery. My framework of choice for some time is the OSGi platform (mostly used on top of Karaf, but I'm getting bolder - going for bare-bone, native containers).

Regarding web applications I admit I don't like sessions and I am strongly inclined to develop stateless applications. Since I like standards and the benefits they provide, my choice for a web framework has narrowed down to JAX-RS for which there are a few implementations.

I came across a project called osgi-jax-rs-connector who's aim is to simplify web application development using JAX-RS on OSGi. The way it works is you write your JAX-RS annotated resources and you publish them in the registry. Once there, the JAX-RS Publisher from osgi-jax-rs-connector will find them, take notice of the annotation and publish them. That's it.

In the project README on github, you will find links to articles detailing the whole process.

All i did was to add a features file for Apache Karaf so you can try it out easily. I've made a pull request with my code to make it part of the original code base and hopefully it will soon.

I'll reproduce the steps below. You start by building the project and installing the features in Apache Karaf:

    feature:repo-add mvn:com.eclipsesource.jaxrs/features/0.0.1-SNAPSHOT/xml/features
    feature:install scr http
    feature:install jax-rs-connector jax-rs-provider-moxy
    install mvn:com.eclipsesource.jaxrs/jax-rs-sample/0.0.1-SNAPSHOT

After this just go to: http://localhost:8181/sercices/greeting

You can check the whole project on my github account in the mean time: step by step .

There are other solution out there for publishing JAX-RS resources using OSGi HttpService. Another interesting approach is Neil Bartlett's JAX-RS OSGi extender . The main advantage (in my opinion) of using the approach taken by Connector is the fact that you publish objects instead of the extender building them for you. This means that I am free to choose the way I build my object and I also have the opportunity to inject dependencies in it before I publish it - hello CDI. I can build my objects using CDI via pax-cdi or with declarative services (as you can see in my sample code) and I am free to inject stuff in it before I expose it for registration with HttpService. That is a pretty powerful thing. I hope to show you how this is done soon.

java mbox parsing with Apache James Mime4j

2012-06-19T13:07:00.000+03:00

Today I just made a small push to Apache James Mim4j trunk. The change-set adds mbox parsing capabilities to mime4j. It's a one class Iterator that you can use to split an mbox file into individual messages and after this parse them with mime4j.

I also added an simple example here.

This is how you can use it:

for (CharBufferWrapper message : MboxIterator.fromFile(mbox).charset(ENCODER.charset()).build()) {

System.out.println(messageSummary(message.asInputStream(ENCODER.charset())));
count++;
}

JETM performance monitoring for Apache James

2012-05-17T12:22:00.000+03:00

The story

Today I decided to improve how people can monitor Apache James email server. James already has some monitoring available via:

Each of the above monitoring options is useful and necessary as it's used to monitor different aspects of how James runs. With regard to performance, only JMX monitoring can provide metrics that allow a user to get a feel of the server state.
With JMX enabled you can just launch jconsole to access the attributes and methods that James exposes: number of active connections active per component, limits, etc. You have complete monitoring access to the following services:

IMAP
POP3
SMTP
LMTP
RemoteManager
DNSService
Queues

The nice thing about JMX is that it's integrated with the JVM (no external libraries also) and can provide very precise information on almost every aspect of your application like memory, execution of methods, number of method calls, etc. You can see what James exposes on the Java Management Extension page. JMX is good so why another solution to monitor James? Well, in order for JMX monitoring to work you have to write some code and once you get it in there it's not easy to turn off. Since James is component based, disabling JMX is very easy: you just have to comment out some spring beans declarations. So the second part is not an issue, but the first one is.
Monitoring all James components means you have to write code for each component. This will make the code-base larger and harder to maintain. Here's where Java™ Execution Time Measurement Library or JETM shines. It's a general library that you can use in your application to do monitoring. It's small, compact, adds little overhead and provides all the basic stuff that you need, including an web interface to view your results.
The nicest part about JETM is that it has spring integration. Because James uses spring extensively (the IOC part) integrating the two was a matter of a few hours (took my time reading).

How it works

The JETM library has integration with spring framework and uses spring AOP. It creates a Proxy to intercept calls to the beans that you wish to monitor.

The result

and detailed:

The code

You won't find much here, just some configurations. You can see the full diff here.
In a few words: add the library to your project pom.xml file:


        <dependencies>
            <dependency>
                <groupId>fm.void.jetm</groupId>
                <artifactId>jetm</artifactId>
                <version>1.2.3</version>
            </dependency>
            <dependency>
                <groupId>fm.void.jetm</groupId>
                <artifactId>jetm-optional</artifactId>
                <version>1.2.3</version>
            </dependency>
        </dependencies>

Next declare the monitoring beans in your project (jetm-monitoring.conf in our case):


    <bean id="etmMonitor" class="etm.core.monitor.NestedMonitor"
          init-method="start" destroy-method="stop" />
    <bean id="etmHttpConsole" class="etm.contrib.console.HttpConsoleServer"
          init-method="start" destroy-method="stop" autowire="constructor"/>

    <bean id="etmMethodCallInterceptor"
          class="etm.contrib.aop.aopalliance.EtmMethodCallInterceptor"
          autowire="constructor"/>

    <bean id="etmAutoProxy"
          class="org.springframework.aop.framework.autoproxy.BeanNameAutoProxyCreator">
        <property name="interceptorNames">
            <list>
                <value>etmMethodCallInterceptor</value>
            </list>
        </property>

Here EtmMonitor is responsible for collecting and aggregating measurements points. The measurements points are method calls intercepted by EtmMethodCallInterceptor. The final step is to add the list of beans from your project that you wish to monitor (also in jetm-monitoring.conf). Here I added most of the beans exported by James.


    <bean id="etmAutoProxy"
          class="org.springframework.aop.framework.autoproxy.BeanNameAutoProxyCreator">
        <property name="interceptorNames">
            <list>
                <value>etmMethodCallInterceptor</value>
            </list>
        </property>
        <!-- add the beans that you wish to monitor to the list bellow -->
        <property name="beanNames">
            <list>
                <value>usersrepository</value>
                <value>recipientrewritetable</value>
                <value>domainlist</value>
                <value>mailrepositorystore</value>
                <!--
            The folowing beans require CGLIB2 to be on the classpath -->
                <!--
            <value>mailqueuefactory</value>
            <value>blobTransferPolicy</value>
            <value>jmsConnectionFactory</value>
            -->
                <value>jmsTransactionManager</value>
                <value>mailprocessor</value>
                <value>mailetcontext</value>
                <value>mailspooler</value>
                <value>mailetloader</value>
                <value>matcherloader</value>
                <value>filesystem</value>
                <value>dnsservice</value>
                <value>fetchmail</value>
                <value>smtpserver</value>
                <value>pop3server</value>
                <value>lmtpserver</value>
                <value>imapserver</value>
                <value>imapDecoder</value>
                <value>imapEncoder</value>
                <value>locker</value>
                <value>datasource</value>
            </list>
        </property>
    </bean>

Spring will do the rest (we import the jetm-monitoring.conf in spring-server.xml to bring it in context).

Final notes

You can do much more than what I've showed. You can export the metrics JETM collects via JMX and console, not just HTTP console. Please read the online docs for more details.
You can get the source code as example by checking out James App . After discussing this on James mailing list a number of other promising monitoring libraries turned up: metrics from Codehale and Servo from Netflix. They both deserve a look.

Trying out (pending) James 3.0-beta4 HBase mail store - take 1

2012-03-04T19:50:00.001+02:00

Today I decided to try the newest (pending) release of James : 3.0-beta4.
I have access to a VM cluster of machines running HBase and I wanted to see how things work, maybe do a small stress test.

Downloaded the binary release from the staging repo, unpacked and began configuring the James to enable the hbase mailstore implementation.

Changed a line in mailbox.conf to enable hbase mailstore provider, changed

import resource="classpath:META-INF/org/apache/james/spring-mailbox.xml"

from spring-server.xml to

import resource="classpath:META-INF/org/apache/james/spring-mailbox-hbase.xml"

and uncommented the mailbox declaration from spring-mailbox-hbase.xml to make the Hbase mail store implementation available to the spring context. After this I copied hbase-site.xml (and hdfs-site.xml) inside the James conf directory.

And with that last step James is configured to send emails to the Hbase cluster. All things done, I started James with bin/run and then.... Exception.

| james.mailprocessor | Unable to init mailet LocalDelivery: org.apache.mailet.MailetException: Could not load mailet (LocalDelivery);
nested exception is:
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'org.apache.james.transport.mailets.LocalDelivery': Injection of resource dependencies failed; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'hbase-mailboxmanager' defined in class path resource [META-INF/org/apache/james/spring-mailbox-hbase.xml]: Cannot resolve reference to bean 'hbase-sessionMapperFactory' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'hbase-sessionMapperFactory' defined in class path resource [META-INF/org/apache/james/spring-mailbox-hbase.xml]: Instantiation of bean failed; nested exception is org.springframework.beans.BeanInstantiationException: Could not instantiate bean class [org.apache.james.mailbox.hbase.HBaseMailboxSessionMapperFactory]: Constructor threw exception; nested exception is java.lang.RuntimeException: java.lang.IllegalArgumentException: Not a host:port pair: �%32649-- removed hostnames-- net,60000,1329515254741
.....

Searched the cause of exception and found out that it's because of a version mismatch. Unfortunately the cluster is running hbase 0.92 and the client is running 0.90. The client serialization is incompatible with the one used by the server.

Aside for the fact that the exception was fatal there is an upside to this: James is trying to connect to the the cluster.

Just found out that HBase 0.92 is on Maven Central, but without tests so this will make upgrading impossible because of the unit tests which require it.

The tests jar will be available in 0.92.1 (as stated on hbase-dev). If that tekes too long I will build and test it with a private release of Hbase 0.92.

Hope things move a bit faster with 0.92.1 with respect to artifacts being available on Maven Central.

NIO Iterator over messages in mbox file

2012-02-08T23:08:00.000+02:00

I've started working on a small project called mbox-iterator that I wish to integrate with mime4j later, when it's more usable.

The idea is to provide an Iterator over all the messages in a mbox file and provide access to the raw data. You can then use mime4j to parse the message and do whatever. It's very good for use cases when you don't need the data to be processed and would like to do your own processing or just need access to the raw data.

The project uses java NIO Memory Mapped files to map the file into memory. This will use the OS to manage the memory and file loading for you you get a ByteBuffer that you can use to accces the date. We are not using this ByteBuffer directly, because it's bytes and we need character sets so we need to decode the bytes. We get use a Charset to get a CharsetDecoder for the encoding we need. CharsetDecoder returns a CharBuffer instance and because it implements CharSequence we can use a regex to determine boundaries between messages.

I was hoping that CharBuffer would share the memory with the MappedByteBuffer that we get initially but it seems that this is not the case. We have some memory copying here because the CharBuffer we get is an instance of java.nio.HeapCharBuffer. I was thinking we could have zero Java heap memory and use just the O.S. pages for holding the mbox data but as it turns out, the process of translating bytes to chars needs some memory to keep the chars.

It would have been nice to use asCharBuffer() and provide zero-copy access, but darn, maybe with a future Java version.

Let's return to our cattle:

When we find such a boundary, we return a slice()ed CharBuffer to that message. We also set position and limit so that we get just the message. This has the advantage of using the same memory as the CharBuffer ww do matching on and avoids unnecessary memory copy operations.

I have tested it on a small mbox (135kb) and performs ok. I'm planning more tests with a 2gb mbox. Using NIO memory mapped files we can map very large files and use the OS cache and buffer memory so we can avoid GC activity and unnecessary memory copy operations.

The following areas need improvements before the project is usable:

- testing for different types of mbox files
- better regex to match different mbox formats (mboxcl, mboxrd and the likes)
- better regex for matching From_ lines (now we miss some From_ lines that have MAILER_DAEMON instead of email address)
- support for mboxes larger than 2GB - use more ByteBuffers to map portions of the files, watch out for mails that spill over boundaries.

I'm pretty new to NIO so if you have suggestions of how to do this better I'm open to suggestions and pull requests.

You will fins a simple example that splits one mbox into individual messages in the project sources.

Happy hacking.

References and links:

[1] http://www.kdgregory.com/index.php?page=java.byteBuffer
[2] http://en.wikipedia.org/wiki/Mbox
[3] http://qmail.org/man/man5/mbox.html
[4] http://james.apache.org/mime4j/index.html
[5] https://github.com/ieugen/mbox-iterator

New job

2011-09-27T17:43:00.000+03:00

A few days ago I joined Big Data Solutions Team at 1and1. They have nice offices in Bucharest, but most of all I like the team and the projects.

Besides the fact that I will continue to work with HBase and Hadoop I will also expand in using Lucene, Solr and Mahout. All Apache technologies :).

I plan to use the experience I gain to integrate more functionality in Apache James, first on my list is mail search over HBase.

James HBase mailbox is in: Give it a spin

2011-09-04T14:36:00.000+03:00

Today I managed to finish integrating the HBase mailbox implemenation for James. I committed all project code based that I developed during Google Summer of code, including the integration tests.

I also added a spring context definition so now it's very easy to use. Just fire up James and supply it (put on the class path) with the xml configuration file for your HBase cluster. Users and domains can also be stored in HBase or in another supported repository (like a database).

This flexibility is achieved because the way James is built. All the important parts are spring components that can be interchanged easily. Also, if you did not already know: James is moving towards OSGI.

So what are you waiting for, go try out the new James goodies on your cluster and send us some feed-back if you have time.

Just remember you have to get the latest trunk version for this and do some compilation. Here you can get detailed instructions on how to do that.

If you are not in a hurry you can wait until we make a new release or check the nightly builds.

Zetar, a free spell-checker dictionary builder, is reborn

2011-09-04T14:09:00.000+03:00

This is a new old project of mine. It's new because I just started to code it and it's old because I had the idea for a very long time (I think I got this idea back in 2006).

The problem:
I believe that there is no good free spell checking dictionary available for Romanian, so I decided to build one.

The solution:
Build a dictionary from published text, based on the frequency of words. The idea is that a bad form will be less common than a good form of the word.

What is available:

I have committed a small spring based project that is able to break text into words and put the words into a database. The schema is very simple, I am counting he number of times each word appears. I plan to further develop this and provide a web interface to access the database. For now, all it does is to scan a XML file (I tested it with the Romanian XML dump of Wikipedia).

It's not very efficient, but it was done in a few hours.
The project uses Spring, Hibernate and Firebird SQL, But you can easily switch another DB if you like.

You can get the sources from my repo: https://bitbucket.org/ieugen/zetar/

Switched Debian to systemd

2011-09-03T21:19:00.000+03:00

Well it happens that I like to live on the edge of technology so from time to time I try something new to satisfy my needs. The last time I switched to Gnome 3, which on Debian means that I have to pull some things from experimental branch.

That's over now and I was thinking what to do next. I was a little unhappy of how my system booted and because I had heard a lot of things about systemd, and all of them good, I decided to try it.

Installation was easy and fast. I followed the instructions available on the Debian wiki page and used aptitude for installation. I am following Debian testing and I also have the unstable branch in my sources.list.

I had just one problem the first time I booted. Gnome 3 (shell) did not load correctly and for a short period of time I had a Gnome2 with UI with a Gnome3 theme. That lasted until I logged out and in again.

Now everything seems to run smooth. My notebook seems to load a little bit faster. I did not run tests to see if it goes up fast, but it definitely goes down fast.

For official things about Systemd, please check the freedesktop project website.

Romanian FLOSS summer school in Arad Castle

2011-09-01T12:52:00.000+03:00

These days I'm participating at an event called "Informatics at the Castle" [1] [2]. It's the third edition and until now I had a wonderful time. I had a talk about the advantages of Free Software and my work at GSoC. I also talked about Apache Software Foundation, Debian, and Ceata.org (a young Romanian FLOSS community) [3].

The presentation is available from here [4].

The participants are from very diverse domains: University teachers (from four Romanian and Moldavian universities), small/medium business owners, students, government agencies employees, LUG members, etc.

That's all for now.

[1] http://www.osor.eu/events/informatics-at-the-castle-free-and-open-source-summer-school/?searchterm=arad
[2] http://proinfo.uvvg.ro/ (in Romanian)
[3] http://ceata.org
[4] http://voinici.ceata.org/~ieugen/files/castiga%20cu%20software%20liber.pdf

Apache James GSoC round-up, HBase mailbox implementation

2011-08-31T15:38:00.000+03:00

It's been a long time since my last post and a lot has happened in the mean time.
I am not a constant writer so I will try to catch up.

A short list of the most important stuff:
- Google Summer of Code is over more or less, and it ended well
- the HBase mailbox implementation is ready, passes all tests, soon to be integrated in the main project
- I was recently elected as a committer for Apache James

So, the great news: James has a mailbox implementation over HBase, and fully functional. To complete the picture, Eric also implemented the user and domain persistence over HBase. You can read more about it here [0]. This makes James fully distributed (when persisting data).

You can find all the code for James HBase mailbox implementation at[1].
I expect to commit the code in the James official repo in the next week, and get update it to follow trunk.

If you are interested, you can check out the mailbox integration tests here [2]. They too will be commited soon.

For my future work I plan to continue working with Jams and improve the implementation (integrate search, and other functionality). So stay tuned, and don't hesitate to ask questions or show interest.

I'm also planning to package James for Debian. Devendra Dave is interested in doing some work in this area. This will help James get some market and better testing and also Debian to have more up-to-date software.

[0] https://issues.apache.org/jira/browse/JAMES-1273
[1] http://code.google.com/a/apache-extras.org/p/mailbox-hdfs/
[2] http://code.google.com/a/apache-extras.org/p/hbase-mailbox-integration-test/

Going to FlossCamp

2011-08-01T23:11:00.000+03:00

It's official: I'm going to FlossCamp this year also. It's the third year in a row.

For those of you who have not heard about FlossCamp. It's the ultimate geek-in-nature getaway and the largest of this kind in Romania.

This year is the fourth edition and will be held at Moeciu, near Brasov in the Carpathians, from 5th to 7th august. There are some pictures of the camping site.

To get an ideea of what to expect you can take a look at the pictures from the previous years.

I had a lot of fun every time, and I know this year is going to be the same.

See you at the camp site it's free (software,) in nature.

Bumped mailbox API version to 0.3

2011-07-03T01:08:00.000+03:00

This is another status update on the James HBase mailbox implementation. I am now following apache-james-mailbox-0.3 release.
At first I tried following trunk, but got overwhelmed by the speed of changes in the code. That led to a discussion on the mailing list that concluded with the me following release 0.2.

Now that the mailbox API is stable in 0.3 I am going to follow this branch. Some of the improvements worth mentioning are:

simpler API for managing messages
lucene index for searching
support for quotas

The full list is available on jira.

The other thing I was working on, the link to HBase, is not done yet. I am getting an exception in HBaseStoreTest.java:



java.lang.NoClassDefFoundError: org/apache/hadoop/security/UnixUserGroupInformation

 at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniDFSCluster(HBaseTestingUtility.java:223)

I think I may have a HBase library that is incompatible with the Hadoop version. I will check this soon and post what I find.

Any suggestions are appreciated.

James HBase mailbox status update: JAXB HBase schema loading

2011-06-27T01:01:00.000+03:00

It's been a long time since my last post. Time to do a status update.

I modeled the HBase implementation after the JPA mailbox implementation. I followed a top-down approach and created most of the classes as stubs that I have to fill. I also created the stub test classes.

Things are not very useful yet because there is no connection to a HBase cluster to test the (not here yet) good stuff. So this is what I'm concentrating on.

We don't have a schema for storing messages in HBase, and I believe it is going to change a few times before we manage to get it right. I figured we need a flexible system to keep up with the changes so I looked for a way to do this. I found an article from Lars George explaining how to change HBase tables from code and it inspired me to write an implementation for James.

I wrote one using Java for XML binding API, JAXB for short.
My implementation has three main classes that map to a hierarchy in XML: ColumnFamily, Table and Store. The classes are stored in org.apache.james.mailbox.hbase.schema package.
A store represents a James storage. A store spans multiple tables (one table for storing messages, another for subscriptions, etc.), each table can contain multiple column families to store different types of information.

While working on this I decided to try Test Driven Development. I found it to be very useful way of testing your design. It really kept me going because after each test passed successfully I got a good feeling. Small successes can mean a lot on the long run. The main benefit of this development method, besides the fact that you can test your code as you go, is that the test you write today can save you (a lot of grey matter) in the future. This is because (good) software that makes it into production is never finished. You always find some bugs to fix, features to add or remove, etc. Your code will change, but it's not always for the best. Some times things break and good tests make the difference.

So now I have code to load a mailbox schema written in XML, I have some test written. The next step is to create a schema and see how these things fit together. This was discussed in part on the mailing list and with my mentor Eric and the discussion summary is on a wiki page.

For the next two weeks I plan to be able to load and create a mailbox schema into HBase and begin testing with different ways of storing messages. I'm thinking of buying „HBase the definitive guide” from Amazon. I just hope a e-book version will be also available.

If you have any comments you can post them on the blog if they are related to the article or on the James mailing list if they are suggestions for James.

See you next time,

First commit to James Mailbox over HBase

2011-06-12T17:31:00.000+03:00

Today I made my first commit to the HBase mailbox implementation for James, the GSoC project I'm working on. It's not much, I just added a simple implementation for MailboxManager and Mailbox interfaces but it feels good. It's what I was preparing for the past couple of weeks.

The above interfaces are used to access mailboxes from the mailbox store. The mailbox is modeled as described in SMTP RFC and Internet Message Format RFC.

In short, a mailbox can be used to refer either:

a character string that represents an email address
a store; a place used to deposit a user's emails

Not all emails hava an associated store, for example email aliases do not.

A more detailed explanation can be found on wikipedia.

For best results check out the latest RFC documents.

The project is hosted at http://code.google.com/a/apache-extras.org/p/mailbox-hdfs/. The sources are managed using mercurial and you can build them using Apache Maven.

References:
[1] RFC 5321, Simple Mail Transfer Protocol, J. Klensin, The Internet Society (October 2008), Section 2.3.11 (Mailbox and Address)
[2] RFC 5322, Internet Message Format, P. Resnick (Ed.), The Internet Society (October 2008), Section 3.4 (Address Specification)

HBase install on Debian

2011-06-12T14:48:00.000+03:00

Today I installed HBase and got the basic exercise. I got the latest version, 0.90.3, directly from HBase home.

I configured it following the Quick start instructions. Basically I instructed HBase to use a directory from a filesystem, instead of a using HDFS cluster. This setup is very good for doing simple examples and getting stuff ready for the real thing.

Installation was not without problems. HBase failed because it could not bind to any network interface. The log file showed the following error:

java.lang.RuntimeException: Failed construction of RegionServer: class org.apache.hadoop.hbase.master.HMasterCommandLine$LocalHMasterCannot assign requested address
at org.apache.hadoop.hbase.util.JVMClusterUtil.createMasterThread(JVMClusterUtil.java:155)
at org.apache.hadoop.hbase.LocalHBaseCluster.addMaster(LocalHBaseCluster.java:197)
at org.apache.hadoop.hbase.LocalHBaseCluster.(LocalHBaseCluster.java:147)
at org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:137)
at org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:102)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:76)
at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:1074)
Caused by: java.net.BindException: Problem binding to /192.168.0.254:0 : Cannot assign requested address
at org.apache.hadoop.hbase.ipc.HBaseServer.bind(HBaseServer.java:203)
at org.apache.hadoop.hbase.ipc.HBaseServer$Listener.(HBaseServer.java:270)
at org.apache.hadoop.hbase.ipc.HBaseServer.(HBaseServer.java:1168)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.(HBaseRPC.java:544)
at org.apache.hadoop.hbase.ipc.HBaseRPC.getServer(HBaseRPC.java:514)
at org.apache.hadoop.hbase.master.HMaster.(HMaster.java:196)
at org.apache.hadoop.hbase.master.HMasterCommandLine$LocalHMaster.(HMasterCommandLine.java:188)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at org.apache.hadoop.hbase.util.JVMClusterUtil.createMasterThread(JVMClusterUtil.java:152)
... 7 more
Caused by: java.net.BindException: Cannot assign requested address
at sun.nio.ch.Net.bind(Native Method)
at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:126)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:59)
at org.apache.hadoop.hbase.ipc.HBaseServer.bind(HBaseServer.java:201)
... 18 more

I first believed the issue was related to IPV6, because I had related problems with java applications accessing the internet on Debian. In that case you can pass the

java.net.preferIPv6Addresses=false

parameter. You can read an explanation here.

After carefully analyzing the log file I noticed that HBase insisted on connecting to a specific IP address. And it wasn't IPv6, it was IPv4 and it had a 0 port ( see the above lising for /192.168.0.254:0).

The documentation states that HBase does a reverse DNS lookup to find it's IP address.
I checked /etc/hosts file and found the culprit. I had a line that linked my machine hostname with the above mentioned IP address.

Changed the address to loopback (127.0.0.1) and that was it: I got HBase to work.

I think the line was added there during the system installation, which is kind of strange because it shouldn't have been there. My Debian installation was clean. I installed Squeeze and then switched to testing.

RLUG meets James

2011-06-10T00:53:00.000+03:00

As I mentioned in a previous post, I did a presentation on James at the monthly RLUG meeting.

It went good, the people seamed interested. My presentation was in Romanian, as the slides and they will be available soon on ProLinux slideshare account. Nicu, a Fedora Ambassador and a friend filmed the event and he will upload the video presentations on blip sometime next week.

I was the first to present so the crowd (about 12 people) was fresh. I pointed out some general things about James because most of the audience where general open-source enthusiasts so I did not go into (that many :D) details. Well I tried. I emphasized the Mailet part and talked about HBase mail storage I am working on.

There where a couple of sys-admins there that managed some mail servers. They pointed out that James seemed more like an enterprise solution, than a solution for someone with more basic needs.

I really did not know how to respond to that. I tend to believe this is not true, but I hope to find out soon (get working experience in this field) and get back to them.

All in all it went well. I left with the impression that I did not bore them. After the presentations we went, as usual, to further discuss the matter at a local „restaurant” (not actually, but I can't find the name for terasă in English).

Last exam for this year, full time to James

2011-06-08T22:34:00.000+03:00

The summer exam session just ended so I will be dedicating my full time to James. The first thing I'm going to do (after I celebrate!) is to write a presentation for the Romanian Linux User Group monthly presentation session.

The RLUG meetings are a monthly event held by the Romanian Open Source Community in Bucharest. It's held regularly from November 2009 and the topics are on open source information technologies.

This is the first time I will be keeping a presentation and I hope to get good feedback from the audience.I will be presenting James -, which I think it's a new technology and I hope some of the people from the audience will experiment with it.

Under RLUG, there is also a periodic workshop event called Atelierele RLUG (eng. RLUG Workshops). They are not as frequent as the meetings. I will prepare to keep James installation, usage and Mailet development workshop. I hope I will be able to pull this off in time for the next event, maybe do some benchmarks betwen James and Postfix or other mail agents.

Stay tuned for updates.

My GSoC project for this year: Apache James mailbox over Hadoop

2011-05-30T15:48:00.000+03:00

The guys at Google where kind enough to select my application for Google Summer of Code program and turn it into a project, which is kind of cool :B.
This is old news but I just started blogging and I am planning to make a habit.

The Project: Mailbox over Hadoop HDFS for Apache James

My project is with the Apache Foundation. I will work on Apache James - Java Apache Enterprise Mail Server - . By the end of this summer I will have to deliver mailbox storage over Hadoop HDFS. Storing email over a distributed file system such as HDFS This will enable James to handle many email accounts and many emails in a

The details and project about the project is handled with Jira.

The people

My mentor is Eric Charles. He's been around with the project for some time and knows his way around it. He helped me get up some inside knowledge. I also met other people on the list (Robert Burrell Donkin and Norman Maurer) and they also seem very passionate.

I think things are going to go well.

The status

It's been a week now since the project officially started. It's been quite busy. I had a lot of new things to learn. The video presentations about Hadoop, HDFS and HBase from Cloudera on vimeo where very helpful.

We discussed on the list and decided to implement the mailbox storage using HBase in order to avoid some performance issues related to the fact that HDFS does not support random file writes. It has only file append feature so once you create a file you can only append to it. This may seem weird for a file system, but you have to keep in mind that HDFS was designed to store huge amounts of data while providing high throughput access to this data. Having random reads and writes is not that easy to achieve if you want to keep the performance high.

Hbase is a NoSQL database implementation. It avoids these issues by appending the changes to a log file in a way journal based file systems do. When the log file gets too large, the changes are committed to the files.

There is a similar mailbox implementation based on Cassandra (another NoSQL store) implemented by IBM. You can find more about it here.

Inmatriculare bulgaria

2009-10-20T12:57:00.001+03:00

Sunt trei modalități de a înmatricula un autoturism în Bulgaria:
-pe numele unei persoane fizice cetățene bulgare,
-pe numele unei persoane juridice bulgare,
-pe numele unei persoane fizice rezidente în Bulgaria.

BgConsult va pune la dispoziție următoarele variante de înmatriculare în Bulgaria:

-Înmatriculare pe numele unui angajat al firmei noastre (roman) rezident în Bulgaria. Dovada proprietății o veți face baza contractului notarial de vânzare-cumpărare și veți circula pe baza procurii notariale în care este specificat că ne reprezentați în dreptul autoturismului în caz de accident, în fața politiei, la radiere, la înmatriculare, plata impozitului, etc.

-Înmatriculare pe numele dumneavoastră pe baza rezidenței pe care noi o vom obține pentru dumneavoastră. Autoturismul va fi permanent în posesia dumneavoastră. Înmatricularea în Bulgaria se va face pe numele dvs. pe baza rezidenței anterior obținute. Cu firma noastră veți face un contract de prestări servicii, prin care noi ne obligăm să vă obținem în Bulgaria rezidența pe 5 ani (durata maximă posibilă, după care trebuie reînnoită) și să înmatriculam autoturismul pe numele dumneavoastră.

-Înmatriculare pe firma noastră din Bulgaria. Dovada proprietății o veți face pe baza contractului de vânzare-cumpărare (perfect valabil în instanță) și veți circula pe baza procurii notariale în care este specificat că ne reprezentați în dreptul autoturismului în caz de accident, în fața politiei, la radiere, la înmatriculare, plata impozitului, etc.

Prețuri:
-Înmatriculare pe numele unui angajat al firmei noastre (român) rezident în Bulgaria 600 euro (contract notarial de vânzare-cumpărare + procura notarială)
- înmatriculare pe firma noastră din Bulgaria 700 euro (contract de vânzare cumpărare + procură notarială)
- înmatriculare pe numele dumneavoastră prin obținerea rezidentei - 650 înmatricularea +300 euro obținerea rezidenței - total 950 euro (ulterioarele autoturisme înmatriculate pe rezidența anterior obținuta vor costa 650 euro)

În oricare din variante este inclusa asigurarea pe un an, impozitul pe un an și toate cheltuielile aferente înmatriculării (și obținerii rezidenței). Prețurile afișate sunt prețurile totale, nicio cheltuială suplimentara nu va interveni.

Proceduri:

-Pentru înmatricularea pe numele unui angajat al firmei noastre (roman) rezident în Bulgaria în principiu durează o singura zi (dacă trimitem cu o zi înainte sa mergem în Bulgaria actele scanate la traducător) în cazul în care intervine ceva sau este aglomerație a doua zi va fi gata cu siguranța. Prezenta dumneavoastră în Bulgaria nu este obligatorie dar binevenita, tine doar de timpul dvs.
-Pentru înmatricularea pe firmă noastră în principiu durează o singura zi (dacă trimitem cu o zi înainte sa mergem în Bulgaria actele scanate la traducător) în cazul în care intervine ceva sau este aglomerație a doua zi va fi gata cu siguranță. Prezența dumneavoastră în Bulgaria nu este obligatorie dar binevenita, tine doar de timpul dvs.
-Pentru înmatricularea pe baza rezidenței va trebui sa procurați "card european de asigurări de sănătate" îl obțineți de la CNAS (Casa Naționala de Asigurări de Sănătate) sau certificat înlocuitor al cardului (se eliberează a doua zi). Vom merge în Bulgaria sa depunem actele pentru obținerea rezidenței și să traducem contractele de vânzare-cumpărare ale autoturismului, după două zile lucrătoare vom merge sa ridicam actele rezidenței și vom înmatricula autoturismul. Dacă intervine ceva sau este aglomerație a doua zi va fi gata cu siguranță. Pentru obținerea rezidenței prezența dvs. în Bulgaria este obligatorie.

Cerințe:
-autoturismele sa nu depășească 120 kw (160.92 cai putere)
-sa fie aduse din UE.

Dacă mai aveți alte întrebări/nelămuriri sau va hotărâți sa optați către una din variante așteptăm răspuns la acest thread sau telefonul dumneavoastră la 0722433194 / 0723712369.
www.bgconsult.ro
www.inmatricularibulgaria.ro
www.rezidentabulgaria.ro
www.firmebulgaria.ro

Cod HTML în blog

2009-07-17T22:01:00.000+03:00

Am întâmpinat o problemă când am scris [0]: codul XSL era interpretat (blogul este HTML) şi nu mai apărea. Mi-am dat seama şi am încercat să repar folosind construcţia

<![CDATA[ ]]

, dar nu a mers (nu prea ştiu HTML), aşa că am căutat altă soluţie: să transform toate elementele pe care HTML-ul le interpretează în codurile echivalente. Am găsit articolul [1] codurile şi cum trebuiesc scrise, dar cum sunt leneş de fel şi nu-mi place să muncesc am căutat o aplicaţie [2]
care s-o facă în locul meu. Am şi găsit un script frumos care transformă cod scris într-o multitudine de limbaje (Java, XSLT, XML, Lua, Perl, Python, Ruby, etc) în HTML sau XML.

[0] http://vorbamutului.blogspot.com/2009/07/referinte-biliografice-in-word-2007-un.html
[1] http://www.blogspottutorial.com/2008/12/html-code-appears-in-posting.html
[2] http://xzfv.appspot.com/s/format.html

Referinţe biliografice în Word 2007 - un coşmar

2009-07-17T20:55:00.001+03:00

Ehe, dacăvreţi să utilizaţi referinţe bibliografice de forma [1], [12] în text (exact ca mai jos) folosind Microsoft Office Word 2007, o să constataţi că nu se poate. Cel puţin nu direct. Pentru a beneficia de asemenea capabilităţi trebuie să faceţi câteva modificări, urmărind paşii de la [0].
Pentru cine nu înţelege engleza, nu are răbdare să caute prin forum, sau prin fişierul XSL nedocumentat de peste 7000 de linii o să scot eu esenţialul mai jos:

Artificiul este posibil modificând un fişier de stil XSL, şi anume "C:\Program Files\Microsoft Office\Office12\Bibliography\Style\ISO690Nmerical.XSL" . Puteţi să-l modificaţi sau faceţi o copie a acestuia pe care să o modificaţi; să-i spunem "ISO690NmericalPatrate.XSL" .
În "ISO690NmericalPatrate.XSL" căutaţi următoarele blocuri :

<xsl:template name="templ_prop_OpenBracket" >
 <xsl:param name="LCID" />
 <xsl:variable name="_LCID">
   <xsl:call-template name="localLCID">
     <xsl:with-param name="LCID" select="$LCID"/>
   </xsl:call-template>
 </xsl:variable>
 <xsl:value-of select="/*/b:Locals/b:Local[@LCID=$_LCID]/b:General/b:OpenBracket"/>
</xsl:template>

<xsl:template name="templ_prop_CloseBracket" >
 <xsl:param name="LCID" />
 <xsl:variable name="_LCID">
   <xsl:call-template name="localLCID">
     <xsl:with-param name="LCID" select="$LCID"/>
   </xsl:call-template>
 </xsl:variable>
 <xsl:value-of select="/*/b:Locals/b:Local[@LCID=$_LCID]/b:General/b:CloseBracket"/>
</xsl:template>

şi modificaţi-le să arate aşa:

<xsl:template name="templ_prop_OpenBracket" >
 <xsl:param name="LCID" />
 <xsl:variable name="_LCID">
   <xsl:call-template name="localLCID">
     <xsl:with-param name="LCID" select="$LCID"/>
   </xsl:call-template>
 </xsl:variable>
 <!--<xsl:value-of select="/*/b:Locals/b:Local[@LCID=$_LCID]/b:General/b:OpenBracket"/>-->
 <xsl:text>[</xsl:text>
</xsl:template>

<xsl:template name="templ_prop_CloseBracket" >
 <xsl:param name="LCID" />
 <xsl:variable name="_LCID">
   <xsl:call-template name="localLCID">
     <xsl:with-param name="LCID" select="$LCID"/>
   </xsl:call-template>
 </xsl:variable>
 <!--<xsl:value-of select="/*/b:Locals/b:Local[@LCID=$_LCID]/b:General/b:CloseBracket"/>-->
 <xsl:text>]</xsl:text>
</xsl:template>

Pentru ca înregistrările bibliografice să arate din aşa:

1. 3-D Constraint-based Modeling: Finding Common Themes. Wiebe, Eric N. 1999. EDGD Mid-year Meeting, Columbus, OH.

aşa

[1]. 3-D Constraint-based Modeling: Finding Common Themes. Wiebe, Eric N. 1999. EDGD Mid-year Meeting, Columbus, OH.

trebuie să încadraţi eticheta

<xsl:value-of select="b:RefOrder"/>

aşa:

<xsl:text>[</xsl:text>
<xsl:value-of select="b:RefOrder"/>
<xsl:text>]</xsl:text>

În ribbonul din word o să apară un stil suplimentar cu același nume (dacă aţi modificat o copie) sau va fi tot stilul ISO 960 (cel cu paranteze rotunde, doar că acum va avea pătrate).

Pentru cei care vor să utilizeze referinţe bibliografice cu paranteze pătrate în Word 2007 le urez succes la muncă (nu cred că aspectul se va păstra şi dacă actualizaţi documentul pe alt calculator, fără să copiati fişierul de stil) şi le dau un sfat: folosiţi OpenOffice.org cu încredere, o să scăpaţi de multe dureri de cap.

[0] http://channel9.msdn.com/forums/Coffeehouse/240589-Word-2007-citations-should-be-simple-but-solved/

curs online java

2009-07-09T13:45:00.000+03:00

M-am apucat să învăț Java și tot căutând materiale pe internet am dat peste un curs gratuit oferit de un domn pe nume Sang Shin. Cursul se numește Free Java Programming with Passion! și îl găsiți la [0] . La sfărșitul cursului veți primi un certificat de absolvire online care o să fie publicat online.

Am început si eu cursul și sunt foarte mulțumuit de modul în care prezintă informația. Este prezentată mai întâi partea teoretică (prezentare și articole sau documentație ) urmată de un laborator cu exemple și o temă pentru acasă.
Este un curs bun, practic, care merită urmat atât de cei care abia învață Java cât și de cei intermediari.

Tot pentru a învăța Java mi-am cumpărat și cartea „Java de la 0 la expert” și pot să spun că prezintă informația destul de bine și ușor de înțeles, deși în unele părți se cam cramponează la detalii. Până acum sunt mulțumit de ea, încă nu am apucat să o termin, dar nu înlocuiește exercițiul. Dacă vreți să învățați Java apucați-vă de scris cod!

Resurse Java:

[0] http://www.javapassion.com/javaintro/
http://java.sun.com/
http://java.sun.com/reference/docs/
http://www.netbeans.org/ - un mediu de programare excelent

Cuplul perfect

2009-07-01T10:57:00.000+03:00

Un individ întreabă un cuplu de bătrânei: "Cum reușiți ca la vârsta asta sunteți un cuplu atât de perfect și va înțelegeți atât de bine?"

El : "Pai, vedeți, TOT ce facem, facem împreună. De exemplu, folosim întotdeauna aceleași prosoape împreuna. Avem unul pe care scrie *C* de la Cap și altul pe care scrie *F* de la Fund.

Ea : "Ceeeeee?!!!!! Întotdeauna am crezut că *C* e de la Cur şi *F* de la Față!