Atif's Blog

Internet of Things (IoT): Raspberry Pi + DS18B20 + Eclipse KURA + Mosquitto + Eclipse Paho MQTT—PART 4

2016-11-16T22:44:00.001+11:00

We need Java applications components to be deployed in Eclipse KURA, to read the readings from DS18B20 via 1-Wire, we have very helpful APIs available to accomplish this, we have famous Pi4J (http://pi4j.com/) library which is specific to Raspberry Pi Platform, the website provides detailed instructions on performing different tasks using the library. After installation of this library, you will find a sample application that shows how to use the library for the 1-Wire protocol.

Here, however, we are not going to use Pi4J library, rather we will use another library for accessing 1-Wire protocol developed by Marcus Hirt. The library includes functionalities for Humidity sensor and Temperature sensor.

In our case, we are only using Temperature Sensor. We are going to use library without any changes, so if you are using Humidity sensor while doing this example, you don’t have to do any additional thing (except that your Humidity Sensor is up and running), this example should be able to collect data from Humidity sensor as well (possibly with a little tweaking as I haven’t tested it)

To utilize the above library, we need to include the jar file (for Temperature sensor) and Shared library (for Humidity sensor) into our component so that they are bundled into the package. With regard to Java code, as it was mentioned earlier, we just made some minor changes in the sample application: Heater that comes with Eclipse KURA while setting up development workspace. I named the project as iot.tempsensor, in Eclipse Workspace looks like below:

There is no noticeable change in code in comparison to the Sample OSGi Bundle: Heater, we are only calling the above-discussed APIs. Like Heater’s bundle, this bundle is also started and stopped through OSGi Declarative Service, which is accomplished through a Component Definition file: ‘Component.xml’ in above example, the file can have any name (not necessarily ‘Component.xml’). We define some important properties in this file and also give the component a name.

Since the component is configurable, we need to provide some initial configuration and allow a mechanism to update the configuration for the bundle at a later stage. To accomplish this, we have to provide a configuration file at location: OSGI-INF>metatype. As per documentation, the name of the file should be the same the component name, hence we named file named as iot.tempsensor.TemperatureSensor.xml. KURA API Interface: ConfigurableComponent documentation says:

“...The ConfigurationService assumes that Meta Type Information XML resource for a given ConfigurableComponent with name abc" to be stored under OSGI-INF/metatype/abc.xml. This is an extra restriction over the OSGi specification: the Meta Type Information XML resource must be named as the name of the Declarative Service Component.”

The Meta Type Information XML resource file (iot.tempsensor.TemperatureSensor.xml) contains two main elements: and . Element: contains an attribute named: id, which takes the value of the Java Class which contains component’s configurable properties. Element: contains an attribute which named: pid which takes the value of Component name defined in Component.xml. Element: contains another element named:

Under Referenced Libraries, w1.jar (For Temperature Sensor) file and libdht.so (For Humidity Sensor) under sharedlib came from Marcur Hirt Library.

See the complete code on GitHub

Once all is done, you may deploy the component on Raspberry Pi running Kura Gateway, please see the instruction on the following link under the section: Remote Target Device:

http://eclipse.github.io/kura/doc/deploying-bundles.html

Upon running the Eclipse KURA framework, the application may be seen in Kura’s Web Interface.

Internet of Things (IoT): Raspberry Pi + DS18B20 + Eclipse KURA + Mosquitto + Eclipse Paho MQTT—PART 3

2016-10-16T04:44:00.000+11:00

Next to complete our environment, we set up an MQTT message broker: Eclipse Mosquitto, on a our development machine or even on a separate machine. There are some publicly available message brokers as well, for example, there is one instance of Mosquitto by Eclipse. IBM Bluemix also includes an instance of MQTT broker.

So Eclipse Kura running on Raspberry Pi, will read the temperature from DS18B20 and sends the data as MQTT message to the broker , MQTT clients, which have subscribed for the topic of the message, will receive the data. Following link describes MQTT and provides a good basic understanding.

https://www.infoq.com/articles/practical-mqtt-with-paho

MQTT primarily belongs Machine to Machine communication and was specifically developed for the purpose, its reliable and resource efficient. I found following resource about the preference of MQTT for Internet of Things (IoT) projects:

https://blogs.sap.com/2015/03/17/why-you-should-use-mqtt-in-iot-projects-and-why-sap-should-support-it/

Since I am working on Ubuntu I just had to do sudo apt-get install mosquitto to install Mosquitto, that’s it!! There are installers for other OSes here: https://mosquitto.org/download/

In my case on Ubuntu, I created a password file under /etc/mosquitto/ using following command:

sudo mosquito_passwd -c mosquito.pwd user_name

This user is used to configure Eclipse Kura

I also set ‘Protocol’ in /etc/mosquitto/mosquitto.conf as following:

protocol mqtt

Try to run Mosquitto using the following comment:

mosquito -c /etc/mosquitto/mosquitto.conf

If it runs fine, you should a similar output as following:

1476551631: mosquitto version 1.4.8 (build date Fri, 19 Feb 2016 12:03:16 +0100) starting

1476551631: Config loaded from mosquitto.conf.

1476551631: Opening ipv4 listen socket on port 9001.

1476551631: Opening ipv6 listen socket on port 9001.

1476551631: Opening ipv4 listen socket on port 1883.

1476551631: Opening ipv6 listen socket on port 1883.

Next we need to configure Eclipse Kura with Mosquitto, instructions for doing so can be found on the following link:

http://eclipse.github.io/kura/doc/cloud-configuration.html

Internet of Things (IoT): Raspberry Pi + DS18B20 + Eclipse KURA + Mosquitto + Eclipse Paho MQTT-P2

2016-10-12T22:47:00.000+11:00

Next we do Eclipse Kura. For an introduction of Eclipse, check following link:

http://eclipse.github.io/kura/doc/intro.html

Eclipse Kura is an OSGi based IoT gateway framework, in our case since Raspberry Pi will be acting as Gateway, therefore, we will install Eclipse Kura on Raspberry Pi. Different versions of Kura have been classified based on Raspberry Pi model. In my case, I chose Raspbian (Model B+).

Notice that another classification is Non-Net and Net(Not having Non-Net as part of those name). Net version provides the facilities to configure the network setting of the gateway device where Kura is running. I first tried Net version and I lost all my network configuration on Raspberry Pi which I didn’t want, so I tried Non-Net version that worked for me and is still running successfully.

In my case I chose Raspbian Model B+ No Net Stable.

Use following link to choose and download a suitable version.

http://www.eclipse.org/kura/downloads.php

The steps for installing Eclipse Kura on Raspberry Pi are listed on the following link, follow all the steps as it is, however, use the debian package downloaded from the above link:

http://eclipse.github.io/kura/doc/raspberry-pi-quick-start.html

If everything goes fine , you should be able to access Kura through browser at http://127.0.0.1:8080 if Raspberry Pi has display. Externally it may be accessed in browser using RasPi’s ip address (on default porti.e. no port required), for example, in my case it was 192.168.1.103.

To develop applications to run on Kura Framework, we need to set up the development environment. Eclipse Kura provides an workspace which contain all the necessary components required. Please see the following link for details:

http://eclipse.github.io/kura/doc/kura-setup.html

There are some tutorials with example applications e.g. HelloWorld Example, Configurable Application. One called Deploying Bundles describes how to install the application (in OSGi terminology called Bundle) on target device in our case its Raspberry Pi.

There is one more tutorial on the same page called Publishing Application, it describes the a demo Application: Heater, its actually this application’s code we will use for our working setup. Pay special attention to this: http://eclipse.github.io/kura/doc/heater_demo.html

Internet of Things (IoT): Raspberry Pi + DS18B20 + Eclipse KURA + Mosquitto + Eclipse Paho MQTT-P1

2016-10-09T21:43:00.000+11:00

These posts describe a working set-up of following component:

Raspberry Pi 3
DS18B20 (1-Wire Digital Temperature Sensor)
Eclipse Kura (Eclipse Project: Framework for IoT Gateway)
Eclipse Mosquitto (Message Broker for MQTT)
Eclipse Paho MQTT (Eclipse Project: MQTT Client)

The post primarily discusses how to make above components work, however, we will try to provide the relevant resources throughout the posts.

We start off with connecting DS18B20 and Raspberry Pi, there are a number of online resources which helps to understand 1-wire sensor and how to connect them to Raspberry Pi, following are the two which I used:

https://www.cl.cam.ac.uk/projects/raspberrypi/tutorials/temperature/

https://cdn-learn.adafruit.com/downloads/pdf/adafruits-raspberry-pi-lesson-11-ds18b20-temperature-sensing.pdf

Above links explains it in a detailed way, we instead look at another issue: in some cases, we come across an error when following command is executed: (discussed on the above links)

cat w1_slave

Error: w1_slave: No such file or directory

Generally when sensor is connected successfully and is working properly, upon executing following commands:

cd /sys/bus/w1/devices/

The list should show your sensor(s) as a directory with its name starting with ‘28-’, the name itself is a combination of characters and numbers. I experienced that in situations where sensor is not correctly working in fact not connected properly, the sensor directory name is although, a combination of characters and numbers but doesn’t start with ‘28-’. In these situation command: cat w1_slave returns the above error.

The source of the error, as mentioned earlier, could be incorrect connections, connecting wires not working properly, incorrect GPIO connections (Caution: As always be careful with GPIO connections).

I have attached some images from my working set-up below. I used 10K ohm resistor in my setup and Dallas Digital 1-Wire Temperature sensor.

Pulse Code Modulation: Quantization

2011-05-27T14:04:00.000+10:00

The discrete signal i.e. sampled values produced as a result of sampling (discussed in the previous post) has to be quantized, hence the process called Quantization. In simple words, this is the process in which a value from a given set of values is assigned to each sample of the discrete signal. The number of values in the set actually is the number of quantization levels to which samples of discrete signals are assigned to. Digital communication is based on bits and bytes, the number of bits used identify the number of quantization levels, hence, in this case when the samples are binary encoded that means they are essentially being quantized in one of the fixed number of quantization levels.

In the figure below, the process has been explained where an input discrete signal s(t) has been quantized into a signal s_q(t). The input signal s(t) moves between low peak amplitude A_L to high peak amplitude A_H . This range from A_L to A_His divided into M intervals (an interval represent a quantization level, also referred as quantization interval) each of size L such that L = (A_H - A_L)/M. In the figure, eight intervals have been shown i.e. M=8. Let A_s be the peak-to-peak amplitude of s(t) i.e. total range between high and low peaks i.e. A_s = A_H - A_Lhence L = A_s /M. The value of each quantization interval is taken as the center of the interval, in figure, shown as a₀ , a₁ , a₂ , a₃ , a₄ , a₅ , a₆ , a₇ . The range of values for each interval has been shown as l₀, l₁, l₂ , l₃ , l₄ , l₅ , l₆ , l₇ . Moreover, the midpoints between consecutive quantization intervals are shown by A₀₁ , A₁₂ , A₂₃ , A₃₄ , A₄₅ , A₅₆ , A₆₇ . Each of these midpoints is at a distance of L/2 from its corresponding intervals values e.g. A₆₇is the midpoint between two quantization intervals whose values are a₆and a₇and is located at a distance of L/2 from a₆and a₇. The distance between two consecutive midpoints and also between two interval values is L. It is, in fact, these mid points which define when the value of the quantized signal s_q(t) changes. It should also be noted that the peak values i.e. A_H and A_L are at a distance of L/2 from their corresponding quantization levels. This results in affecting the peak-to-peak amplitude of s_q(t) shown as A_q and has a value of (M-1)L.

As shown in the figure, when the signal being quantized i.e. s(t) is in the range of l₇ , the quantized signal s_q(t) maintains a constant value of a₇ . As soon as s(t) gets in the range of l₆ , s_q(t) makes a jump to the value of a₆and maintains this value as long as s(t) assumes a value in the range of l₆ . The sudden jump is made when s(t) crosses the midpoint A₆₇. Similar changes occur to s_q(t) through the rest of the process of quantization. Therefore, at any time, the quantized signal s_q(t) holds a value from a₀ , a₁ , a₂ , a₃ , a₄ , a₅ , a₆ and a₇ . That means, at any moment, s_q(t) exits in one of the quantization intervals that is nearest to s(t).

As the above description and the figure indicate that s_q(t) is not same as s(t) rather it’s an approximation with a difference between the two. This difference is very important and is known as Quantization Error. Quantization Error’s magnitude is always less than or equal to L/2 (We’ll discuss Quantization Error later). The approximation can be improved by making the interval size smaller, hence increasing the number of intervals (quantization levels). In digital communication, high number of intervals requires high number of bits which itself, in turn, require high transmission bandwidth. On the other hand, if interval size is increased using lesser number of quantization levels, approximation is poor. Therefore, there has to be a tradeoff between approximation and bandwidth.

Pulse Code Modulation: Filtering and Sampling

2011-03-17T18:58:00.000+11:00

In Digital Telephony, analog signals have to be converted into digital signals before transmission because of several advantages they have over Analog signals, one of which is their strong immunity against noise. Pulse Code Modulation (PCM) is one of the techniques to achieve the objective. Generally, PCM is a four step process consisting of Filtering and Sampling, Quantization and Encoding. However, Quantization may be performed either of two ways: Linear Quantization or Non-Linear Quantization. For Non-Linear Quantization, sometimes an additional step of Companding is performed after Sampling and before Quantization to prepare the samples for non-linear Quantization.

Here, in this post, we’ll discuss Filtering and Sampling. Later, in the next posts, we’ll discuss Linear Quantization, Quantization Error, Non-Linear Quantization, Companding and Encoding. Our discussion is based on digital telephony.

Analog Low Pass Filtering:

Before an analog signal is converted into digital one, it is passed through an analog Low-Pass Filter (see here). Following figure shows the frequency responses of a Low Pass Filter.

This filtering comes handy later to avoid aliasing when the analog signal is sampled and then reconstructed from the digital signal (see here), hence also called Anti-Aliasing Filter. In simple words, this is performed to filter out the frequencies which are higher than a “certain frequency”. We are interested in the frequency band lower than the “certain frequency” and not the higher.

“Certain Frequency” is just a made up term for explanation and is not a technical term.

For Telephony Systems whose main objective is to transfer human voice, that “certain frequency” can be found from high school physics. We have read human voice consists of frequency range of 300Hz to 3000Hz. Due to some constraints in designing an ideal filter, we make this certain frequency 4000Hz instead of 3000Hz. So after filtering, for the next step of Sampling, we know that bandwidth of the signal is 4000Hz i.e. the greatest frequency analog signal consists of is 4000Hz.

Sampling

Sampling is the process of creation of a Discrete Signal by taking samples of an Analog Signal. These samples are taken at regular intervals; intervals may be time base (for time domain) or space based (for spatial domain) or any other independent variables. In our case, intervals are generally time based, these regular intervals define the sampling rate--the rate at which samples are taken, also called sampling frequency F_s.

Sampling is governed by Nyquist Sampling Theorem, according to which when a signal is sampled, the sampling rate (or sampling frequency) must be at least twice the highest frequency the signal contains, to avoid aliasing. In other words, sampling frequency must be at least twice the bandwidth of the signal. This means at least two samples are required per cycle of each frequency in the analog signal.

Therefore

F_s = 2 x 4000 = 8000 Samples/sec

Filtering and Sampling work in collaboration to overcome a possible artifact of aliasing which is a result of under sampling. Aliasing is an important concept to be understood and an important artifact to be overcome for correct transfer of voice data in telephony systems.

On the other hand, sampling rate, higher than minimum sampling rate of 8000 samples/sec, results in too much voice data to be transferred over the communication channel demanding a high bandwidth. Between the two extremes of under-sampling and over-sampling, above sampling frequency of 8K samples/sec is the threshold which is good enough for the reconstructed analog signal and is good enough for the usage of bandwidth.

A technique often used for sampling is Pulse Amplitude Modulation (PAM) in which a pulse is modulated with the filtered analog signal. This has been shown is the following figure created on Simulink Matlab, pink colored graph is the pulse, yellow colored graph is a sine wave to be sampled and aqua graph is the output of the PAM sampling.

Sampling creates discrete signals not the digitized signals; Quantization is the step to digitize the samples into binary encoded form.

Probability Density Function (pdf) of Sine Wave

2011-02-21T22:02:00.000+11:00

Probability Density Function, f(x), completely describes the probability that a Random Variable, X, assumes a value in a given range of possible values.

Mathematically, the probability that random variable X, assumes a value that lies in the interval [a,b].

The above equation represents the area under the curve f(x) from a to b. The limits of the above integral depend upon the domain of the f(x). However, the total area under the curve must not be greater 1 whatever the limits are. That is

Another jargon often used with Probability Density Function is Cumulative Distribution Function (cdf). As the name implies it is the cumulative probability that a random variable X takes on a value equal or less than some specific value. For given value a, F(a) is the Cumulative Distribution Function that shows the probability that the random variable X assumes a value equal to or less than a. Mathematically shown as below

P(X < a) = F(a) ………….(3)

Cumulative Distribution Function is based on probability, it has following properties:

0 <= F(a) <=1

F(a₁) <= F(a₂) if a₁< a₂

F(-∞) = 0

F(+∞) = 1

The relationship between Cumulative Distribution Function (cdf) and Probability Density Function (pdf) is shown below

That means the value of the F(a) is equal to the area under probability density function curve upto a.

From the above relationship, conversely, we have

Above two equations give us the relationship between cdf and pdf.

Following links provide a very simple understanding of pdf and cdf:

Engineered Software, Inc. (pdf)
Engineered Software, Inc. (cdf)
Wikipedia

Now for Probability Density Function of a sine wave. Let’s say we have

X = sin(Y) ………………… (6)

A sine wave has been shown in figure below:

For equation (6), we already know that Y is uniformly distributed over [- pi/2 , pi/2] so X takes on a value in the range [-1, 1]. In this case, X is our random variable for which probability density function is given by eq (1). We aim to find the function f(x) for the sine wave given in eq (6).

Generally the inverse of a function provides the distribution of the values of that function which can be converted into Cumulative Distribution Function (cdf) after a little manipulation. These manipulations are done so that it fits with the definition of cdf.

We have random variable X which can assume a value in the interval [-1, 1]. Therefore

P(X < a) = P (sin(Y) < a) = P(Y < arcsin(a)) where -1<=a<=1

Hence

Notice the modifications of dividing factor of and additive factor of ½ which have been done to make it compliant with the definition of Cumulative Distribution Function and its properties shown above.

Cumulative Distribution Function of a sine wave is shown below:

The possible values that a sine function can assume are shown along X-axis while probability is shown along Y-axis. The maximum probability can be 1.

Now making use of equation (5) above, we can calculate the Probability Density Function (pdf) of sine function (sine wave) by taking the derivative of equation (7). Hence pdf of sine wave is given below

This is quite intuitive that as the slope of a function increases at a value, the chances of occurring that value are higher. (Think of a straight line parallel to Y-axis which has infinite slope and has only one value occurring all the times along X-axis). The probability density function (pdf) of a sine wave, f(x), is shown below:

The pdf shows that probability is high near the extreme values of -1 and 1 as most of the possible values occur towards the extreme values. For example, sine function reaches at 0.5 (half of its max amplitude) just at pi/6 (i.e. 30 degrees).

All the above plots have been created from rechneronline.

Asterisk with Oracle

2011-01-09T02:43:00.000+11:00

...continuation

Now Asterisk:

Having installed unixODBC and unixODBC-devel, I download Asterisk and installed it.

./configure

make or make menuselect

make install

During the execution of ./configure script, on finding uinxODBC and unixODBC-devel installed, it prepares all ODBC related modules for installation, it can be verified it by using ‘make select’ instead of ‘make’. After ‘make install’ had finished, my Asterisk was ODBC enabled, all I needed to do was configure the conf files.

Before changing the conf files, I created a table called ‘sip_buddies’ in my database, schema can be found here, I had to make some changes in the schema according to my database. I was planning to test peer registration from database.

On Asterisk, I was interested in dynamic mode of Asterisk Realtime Arcitecture (ARA), based on that, I had to change the following conf files:

/etc/asterisk/modules.conf

/etc/asterisk/res_odbc.conf

/etc/asterisk/extconfig.conf

In modules.conf, I uncommented following two lines:

preload => res_odbc.so

preload => res_config_odbc.so

In res_odbc.conf, I added following context:

[asterisk_oracle]

enabled => yes

dsn => OracleODBC-11g ; the context added to odbc.ini

;username => testdb ;already provided in odbc.ini , may be provided here too

;password => testdb ;already provided in odbc.ini, may be provided here too

pooling => no

limit => 0

pre-connect => yes

Like username and password, there are other parameters in odbc.ini which may be overridden here in this file, as has been mentioned at the above link.

Finally, I changed the extconfig.conf and added the following line:

sippeers => odbc, asterisk_oracle, sip_buddies

‘asterisk_oracle’ is the database as I created in res_odbc.conf. sip_buddies was the table in my database where peers resided.

After running asterisk, I executed following command on CLI

odbc show

The output showed that I was connected to OracleODBC-11g.

Then I added some peers in my peers table while Asterisk was running and tried to register them on Asterisk through Soft Phones.

Note: I had a little problem here while registering the peer with following output:

chan_sip.c:20603 handle_request_register: Registration from…….. failed for…….. - No
matching peer found.

It was fixed by populating the ‘defaultip’ field of my table sip_buddies, a null value in this field was creating the problem. This problem has been discussed with a patch that is available here. I hope the future release of Asterisk would have it fixed or you can do it yourself from above link by changing the source.

It’s all worth it. I was successfully able to register my peers without a reload. Everything I did was working fine especially about unixODBC-devel which seemed like a nightmare!!!!!!

You can do more things with this, I hope this long post (splited into four parts) is helpful to all you guys who dealing with the same issue. Let me know, if it is useful for you.

Asterisk with Oracle

2011-01-08T23:38:00.000+11:00

...continuation

Put it all Together:

As a summary of above discussion, l used following components:

unixODBC 2.3.0 (available here)
Oracle Instant Client 11g 11.2.0.2-both Basic and ODBC Supplement Packages (available here)
unixODBC-devel 2.2.11 (Initially unixODBC 2.2.11 is also needed to identify the inter-dependency between both)

As said before, I’ll be using absolute paths for referring directories and file in this discussion. Also I executed all the commands as super user—root.

Following command installed unixODBC 2.2.11 and unixODBC -devel 2.2.11 from repository (you may install them by downloading rpm packages from internet).

yum –install unixodbc unixodbc-devel

Files related to both of these packages resided in different directories after installation which can be check using the commands ‘rpm -ql unixodbc-devel’ and rpm ‘rpm -ql unixodbc’. Here I was merely interested in identifying the dependency of unixODBC-devel on unixODBC. Apart from other files, both packages had their Static and Shared library files in /usr/lib/ directory. Some of the Shared library files of unixODBC-devel were actually soft links to the Shared library files of unixODBC which formed the dependency on unixODBC. I was interested only in these shared library files of unixODBC, others had no importance. My plan was to provide unixODBC-devel with these shared library files from unixODBC 2.3.0. That was the only purpose of installing unixODBC 2.2.11 and was supposed to be uninstalled after it had served its purpose. But before uninstalling it I decided to make a copy of all of its files especially shared library files, so that I could use them if I didn’t find any of the shared libraries from unixODBC 2.3.0. I used the following Shell script to copy all files of unixODBC 2.2.11 in a directory /usr/backup/ unixODBC/.

#!/bin/sh

for file in `rpm -ql unixODBC`

do

     cp "$file"  /usr/backup/unixODBC/

done

Now I could uninstall unixODBC 2.2.11 using command (yum –e unixodbc) but as unixODBC-devel had a dependency on unixODBC, Package Manager asked to remove unixODBC-devel too which I needed for Asterisk. I used the following command to remove unixODBC package forcing the package not to check the dependencies.

rpm –e --no-deps unixodbc

This removed the unixODBC package successfully but this made following soft linked shared library files of unixODBC-devel unstable/corrupted having no source file in the directory /usr/lib/ (try ls -lrt). As mentioned before, I had planned to provide these library files from unixODBC 2.3.0.

UnixODBC-devel 2.2.11	UnixODBC 2.2.11
libtemplate.so	libtemplate.so.1.0.0
libtdsS.so	libtdsS.so.1.0.0
libsapdbS.so	libsapdbS.so.1.0.0
liboraodbcS.so	liboraodbcS.so.1.0.0
liboplodbcS.so	liboplodbcS.so.1.0.0
libodbctxtS.so	libodbctxtS.so.1.0.0
libodbcnnS.so	libodbcnnS.so.1.0.0
libodbcminiS.so	libodbcminiS.so.1.0.0
libodbcinst.so	libodbcinst.so.1.0.0
libodbcdrvcfg2S.so	libodbcdrvcfg2S.so.1.0.0
libodbcdrvcfg1S.so	libodbcdrvcfg1S.so.1.0.0
libodbccr.so	libodbccr.so.1.0.0
libnn.so	libnn.so.1.0.0
libmimerS.so	libmimerS.so.1.0.0
libgtrtst.so	libgtrtst.so.1.0.0
libesoobS.so	libesoobS.so.1.0.0
libboundparam.so	libboundparam.so.1.0.0

Next I downloaded and put unixODBC-2.3.0.tar.gz in the directory /usr/local/src/ and installed it by executing the following set of commands:

tar zxvf unixODBC-2.3.0.tar.gz (This created a directory named unixODBC-2.3.0)

cd unixODBC-2.3.0

./configure --prefix=/usr/

make

make install

This installed the unixODBC-2.3.0’s binary files in /usr/bin/, header files in /usr/include, ini files (odbc.ini, odbcinst.ini) in /usr/etc/ and shared library files in /usr/lib/. The prefix (--prefix=/usr/) passed to the ‘configure’ script purposely so that shared libray files of unixODBC 2.3.0 are put in /usr/lib/ directory to resolve the unstable linked shared library files of unixODBC-devel 2.2.11.

I noticed there were only two files (libodbcinst.so.1.0.0 and libodbccr.so.1.0.0) from unixODBC 2.3.0 which matched two of unstable/corrupted shared library files of unixODBC-devel (libodbcinst.so and libodbccr.so); highlighted in yellow in above table. For the rest of the files, I copied the original shared library files of unixODBC 2.2.11 of which I had made a backup in /usr/backup/ unixODBC/ previously. After copying these shared library files, all unstable files of devel were stable now. (Again try ls -lrt).

unixODBC can be checked by through following command:

odbcinst –j

Next I downloaded Oracle Instant Client 11g, both Basic and ODBC Supplement packages and unzipped both of them in same directory /opt/instantclient_11/.

/opt/instantclient_11/ directory contains a script called odbc_update_ini.sh. This script takes four arguments, three of which are optional. I passed only the compulsory argument and leave the optional arguments. The only argument I sent (the one which is mandatory) is the path to Driver Manager that simply means the prefix of etc/odbc.ini and etc/odbcinst.ini.

As I mentioned before when I installed unixODBC 2.3.0, it put ini files in /usr/etc/ so I executed the above script as below:

cd /opt/instantclient_11/

./odbc_update_ini.sh ‘/usr/’

This script automatically configures the Oracle Driver with unixODBC 2.3.0 by updating two files; /usr/etc/odbcinst.ini and /root/.odbc.ini. (If non-super user is logged in while executing this script, then /home/username/.odbc.ini is update)

In /usr/etc/odbcinst.ini, I had an entry like the one below:

[Oracle 11g ODBC driver]

Description = Oracle ODBC driver for Oracle 11g

Driver = /opt/instantclient_11_2/libsqora.so.11.1

Setup =

…………….

In /root/.odbc.ini, I had an entry like the one below:

[OracleODBC-11g]

Application Attributes = T

…………….

Driver = Oracle 11g ODBC driver

DSN = OracleODBC-11g

…………….

……………

I edited the above entry in /root/.odbc.ini, I added some new fields, commented some existing fields (using ;) and changed the values of some of the fields. After these changes its looks like the below:

[OracleODBC-11g]

Application Attributes = T

Attributes = W

BatchAutocommitMode = IfAllSuccessful

BindAsFLOAT = F

CloseCursor = F

DisableDPM = F

DisableMTS = T

Driver = Oracle 11g ODBC driver ; the entry from odbcinst.ini

DSN = OracleODBC-11g

EXECSchemaOpt =

EXECSyntax = T

Failover = T

FailoverDelay = 10

FailoverRetryCount = 10

FetchBufferSize = 64000

ForceWCHAR = F

Lobs = T

Longs = T

MaxLargeData = 0

MetadataIdDefault = F

QueryTimeout = T

ResultSets = T

Database = XE ;db name

ServerName = xxx.xxx.xxx.xxx ; db ip

Port = 1521 ;db port

;ServerName = 192.168.0.105:1521/xe ; has been commented, may be used as well

SQLGetData extensions = F

Translation DLL =

Translation Option = 0

DisableRULEHint = T

;UserID = testdb ;has been commented

UserName = testdb ;db user

Password = testdb ;db password

StatementCache=F

CacheBufferSize=20

UseOCIDescribeAny=F

I copied this entry i.e. [OracleODBC-11g] in /root/.odbc.ini to /usr/etc/odbc.ini, so both ini files have the entry [OracleODBC-11g] with same fields and their values.

.odbc.ini (with a dot) is read when db connection is being made by the user himself or herself while odbc.ini (without a preceding dot) is read when a db connection is being made by an application such as Asterisk. (Try odbcinst –q –d to see the driver’s details)

I executed the following command of isql, which is a utility application shipped with unixODBC, to check my Oracle DB connection.

isql –v OracleODBC-11g

I got the following output showing that connection has been successful with the DB.

+---------------------------------------+

| Connected! |

| |

| sql-statement |

| help [tablename] |

| quit |

| |

+---------------------------------------+

This successful connection does not mean that what I did for unixODBC-devel is okay, that could only be confirmed if Asterisk was successful in communicating with the databse. I could make a successful connection with database using isql from unixODBC 2.3.0 without even installing unixODBC-devel. However, unixODBC 2.2.11 wasn’t successful in making a connection with database through isql because that doesn’t work with Oracle Instant Client.

to be continued

Asterisk with Oracle

2011-01-08T22:18:00.000+11:00

....continuation

Note: As I mentioned previously, this is not a step by step guide of integration of Asterisk with Oracle, I am just writing down everything which I did to make this integration work, which means some of the things I did may not be necessary i.e. there might be some extra steps I did. I didn’t have the opportunity and the time to identify which steps are exactly required. Also in this discussion, I am using absolute paths to referring to directories and files which I had on my system. You can adjust your paths accordingly or just use the ones I did, they are standard places.

Problems and Solutions:

Asterisk works with any RDBMS through ODBC. I was working on Linux CentOS 5, so I used unixODBC—an ODBC implementation for UNIX based Operating Systems. For more details see here. unixODBC (or any implementation of ODBC) needs a driver to communicate with the RDBMS e.g. in my case, I needed Oracle Driver for unixODBC that allows it to communicate with Oracle. There is one Oracle Driver available from easysoft but for this you have to purchase the license- A Problem!!!!!!

So I decided to use Oracle Instant Client with two of its packages—Basic and ODBC Supplement (see here). These two together solve my first problem of a free Oracle Driver. However, somewhere on the web (I can’t remember the link), I read that Oracle Instant Client 10g works only with specific distros of Linux and CentOS wasn’t there in the list-Another Problem!!!!

For which I opted to use Oracle Instant Client 11g—ver:11.2.0.2.0 (available here), this solves both of my problems.

As I said earlier, Asterisk needs unixODBC (an implementation of ODBC), it also needs unixODBC-devel to communicate with the RDBMS. You have to have both these packages installed on your Linux box before you make/install Asterisk if you intend to make Asterisk communicate with a RDBMS. On executing ‘yum’ command to install both these packages (yum install unixODBC unixODBC-devel), I was able to install unixODBC 2.2.11 and unixODBC-devel 2.2.11 packages. The worst thing I came to know: Oracle Instant Client does not work with unixODBC 2.2.11, it needs unixODBC > 2.2.11 and the latest available on CentOS’s repository was 2.2.11. (I also checked at Ubuntu’s Repository on another machine, it was same as CentOS)-Yet Another Problem!!!!

Okay fine!!! I decided to use unixODBC 2.3.0 available here, this fixes my problem but I didn’t find the corresponding devel package anywhere on the web (I was assuming it would be something like unixODBC -devel 2.3.0)-Problem after Problem!!!

After doing quite a lot search for some work around, I decided to go with unixODBC -devel 2.2.11 and unixODBC 2.3.0. I tried to install unixODBC -devel 2.2.11 using ‘yum’ command (yum install unixODBC -devel). This also tries to install unixODBC 2.2.11 showing there is a dependency of unixODBC -devel upon unixODBC -PROBLEMS!!!!!!!!!

Now the new task I had was to identify the dependencies of unixODBC -devel 2.2.11 upon unixODBC 2.2.11 and map them to unixODBC 2.3.0 instead of unixODBC 2.2.11. That means I was going to use unixODBC 2.3.0 and satisfy unixODBC -devel 2.2.11 from unixODBC 2.3.0 for what it needed from unixODBC 2.2.11. If there still are dependencies unresolved from unixODBC 2.3.0, I resolved them through unixODBC 2.2.11 as you see later.

Luckily, this proved to be the last problem that I had to resolve. Following discussion describes how I actually did all the above things.

to be continued ...

Asterisk with Oracle

2011-01-07T00:43:00.000+11:00

Asterisk works with its conf files where it finds the information it needs. At start up, Asterisk loads all the information from those files, now if you make a change in any of the conf, you have to either restart Asterisk or reload the module the file belongs to. This is our first situation, let’s call it Scenario A.

Now think about a practical deployment of Asterisk where you want do some sort of load sharing having multiple instances of Asterisk. However, you want to have same configuration for all instances of Asterisk e.g. dial plans, peers, users etc. You will have to do the tedious job of updating the conf files for each instance of Asterisk; the situation worsens if you have to make a change in any of the conf files. This is our second situation, let’s call it Scenario B.

Both of these scenarios are not a big problem, if they don’t have to be performed frequently but there are some files which might be required to be updated frequently e.g. changes in dial plans, adding/removing/updating peers or users, these files includes extensions.conf, sip.conf, iax.conf, users.conf etc.

Well scenario A may be resolved by having someone reloading the conf files after the changes, or you may write an external script (e.g. Shell Script) using a command something like asterisk reload and have that script running on regular basis in a scheduler e.g. crontab. However, that doesn’t sound like a reasonable solution, moreover, it doesn’t fit with our Scenario B.

How about putting this configuration information in an external database and having Asterisk to load that information from the database rather than files. That’s solves our problem in scenario B, put the configuration in one database and configure all instances of Asterisk to load the configuration from a single place--Database. Asterisk is equipped with the capability/feature to read the configuration from database through a feature called Asterisk Real-time Architecture (ARA). You can find more information about ARA here. Or you can find this information in ‘doc’ directory of untarred Asterisk directory. ARA mainly works in two modes: Static Real-time and Dynamic Real-time. Static Mode works like the plain conf files i.e. configuration is loaded at the startup and if you make changes, you have to do a reload; asterisk loads the configuration from the database. On the other hand, Dynamic Real-time works in dynamic way, it does load the information at the startup from database but if you make a change in database, it automatically reflects on Asterisk, that means NO RELOAD. Isn’t it great? For example, you add a sip user to database through a front end interface; it gets provisioned automatically on Asterisk!!!!! That fixes our problem for scenario A.

You can find ample information on ARA and its configuration with MySQL and PostgreSQL on the internet and Asterisk’s free book available here. I’m going to discuss Asterisk’s integration with Oracle. My discussion is more a kind of itinerary rather than a guide towards the issue, it describes what problems I faced with and what did I do to resolve them.

to be continued...

FreeRADIUS with Oracle

2011-01-06T02:09:00.000+11:00

Recently working on FreeRADIUS Sever v2.1.10, I found that there are some problems in making FreeRADIUS to work with Oracle. I had the same problem described at here. Following is the problem statement put by Alexandre, at the above link: (Great work Alexandre!!)

Unless I missunderstood the compil process it seems to me that todays source tree is stuck to Oracle 10g. Indeed the configure.in file for the rlm_sql_oracle module has a library name harcoded: libnnz10.
If you run Oracle instantclient 11g (I guess it's the same with a full installation of Oracle 11g) the library is named libnnz11. As a result, oracle linking test wiil always fail even with a correct --with-oracle-lib-dir set.
To be able to compile with both version of Oracle (10g and 11g)
I suggest a new option can be set that would define the version during ./configure.
If it sounds ok for everyone and can be tested by more people (I only tested it on Debian with instantclient_11.1), here is "works for me" patch that introduce the --with-oracle-version switch.

As Alexandre rightly guessed, there exists the same problem with full installation of Oracle 11g. However I reckon there shouldn’t be any problem with Oracle Instant Client 10g or Full Oracle installation of 10g when being configured with FreeRADIUS v2.1.10 and v2.1.9.

Although, required changes have been mentioned at the above link, I am going to provide the complete file here, so you don’t have to make the changes yourself and everything is in place before you start installing FreeRADIUS. This will work for 10g and 11g both instant client and full installation. By the way, I have tried this on Red Hat and CentOS 5. While trying on CentOS, I was trying to install FreeRADIUS Sever v2.1.9 with full Oracle 11g Installation and was struck with the same problem.

Once you have downloaded and untarred the FreeRADIUS tar file. (My untarred FreeRADIUS directory is in /usr/local/src i.e. /usr/local/src/freeradius-server-2.1.10). In this directory, go to ./src/modules/rlm_sql/drivers/rlm_sql_oracle directory. So the complete path becomes /usr/local/src/freeradius-server-2.1.10/src/modules/rlm_sql/drivers/rlm_sql_oracle. Replace the contents of ‘configure.in’ file with the contents I have provided here. (I can upload this file to some location if you need it and if you let me know some place)

AC_INIT(sql_oracle.c) AC_REVISION($Revision: 1.10 $) AC_DEFUN(modname,[rlm_sql_oracle]) fail= sql_oracle_ldflags= sql_oracle_cflags= if test x$with_[]modname != xno; then AC_MSG_CHECKING([for oci.h]) dnl # dnl # See if the user passed in the oracle_home_dir option and dnl # use that first. If not, use $ORACLE_HOME. If that's dnl # not defined, give up. There's no point in blindly dnl # hunting around for Oracle - there's no standard place dnl # for it. Any sane Oracle user/developer should have $ORACLE_HOME dnl # defined anyways. dnl # dnl ############################################################ dnl # Check for command line options dnl ############################################################ dnl extra argument: --with-oracle-version=VER oracle_version= AC_ARG_WITH(oracle-version, [AS_HELP_STRING([--with-oracle-version=VER], [Version of Oracle to search for. Should be 10 for Oracle 10g and 11 for 11g])], [case "$withval" in 11) oracle_version="$withval" ;; 10) oracle_version="$withval" ;; *) AC_MSG_ERROR(Need oracle-version) ;; esac]) dnl extra argument: --with-oracle-include-dir=DIR oracle_include_dir= AC_ARG_WITH(oracle-include-dir, [AS_HELP_STRING([--with-oracle-include-dir=DIR], [Directory where the Oracle includes may be found. It should be located under ORACLE_HOME if you have client SDK installed. Use --with-oracle-include-dir=yes if you are sure that your compiler include path includes Oracle include dir. Hint: you might try to do: locate oci.h])], [case "$withval" in no) AC_MSG_ERROR(Need oracle-include-dir) ;; yes) ;; *) oracle_include_dir="$withval" ;; esac]) dnl extra argument: --with-oracle-lib-dir=DIR oracle_lib_dir= AC_ARG_WITH(oracle-lib-dir, [AS_HELP_STRING([--with-oracle-lib-dir=DIR], [Directory where the oracle libraries may be found. It should be located under ORACLE_HOME. iUse --with-oracle-lib-dir=yes if you are sure that your linker will find the necessary Oracle client libs. Hint: you might try to do: locate libclntsh.so])], [case "$withval" in no) AC_MSG_ERROR(Need oracle-lib-dir) ;; yes) ;; *) oracle_lib_dir="$withval" ;; esac]) AC_TRY_COMPILE([#include ], [ int a = 1;], ORACLE_INCLUDE=" ", ORACLE_INCLUDE= ) RLM_SQL_ORA_WORKING_CFLAGS_FOR_LINKING_TEST= if test "x$ORACLE_INCLUDE" = "x" ; then old_CFLAGS="$CFLAGS" CFLAGS="$old_CFLAGS -I$oracle_include_dir" AC_TRY_COMPILE([#include ], [ int a = 1;], ORACLE_INCLUDE="-I$oracle_include_dir", ORACLE_INCLUDE= ) RLM_SQL_ORA_WORKING_CFLAGS_FOR_LINKING_TEST="$CFLAGS" CFLAGS="$old_CFLAGS" fi # Proceed to linking makes only sense if include dir is OK. if test "x$ORACLE_INCLUDE" != "x" ; then old_LIBS="$LIBS" old_CFLAGS="$CFLAGS" CFLAGS="$RLM_SQL_ORA_WORKING_CFLAGS_FOR_LINKING_TEST" ORACLE_LIBDIR_SWITCH= if test "x$oracle_lib_dir" != "x" ; then ORACLE_LIBDIR_SWITCH="-L${oracle_lib_dir} " fi if test "x$oracle_version" = "x" ; then AC_MSG_RESULT(no) AC_MSG_WARN([oracle version not found. Use --with-oracle-version={10|11}.]) fail="$fail Oracle version" else LIBS="$old_LIBS $ORACLE_LIBDIR_SWITCH -lclntsh -lnnz${oracle_version}" AC_TRY_LINK([#include static OCIEnv *p_env; static OCIError *p_err; static OCISvcCtx *p_svc; static OCIStmt *p_sql; static OCIDefine *p_dfn = (OCIDefine *) 0; static OCIBind *p_bnd = (OCIBind *) 0; ], [ int p_bvi; char p_sli[20]; int rc; char errbuf[100]; int errcode; rc = OCIInitialize((ub4) OCI_DEFAULT, (dvoid *)0, /* Initialize OCI */ (dvoid * (*)(dvoid *, size_t)) 0, (dvoid * (*)(dvoid *, dvoid *, size_t))0, (void (*)(dvoid *, dvoid *)) 0 ); ], ORACLE_LIBS="$ORACLE_LIBDIR_SWITCH -lclntsh -lnnz${oracle_version}", ORACLE_LIBS= ) LIBS="$old_LIBS" CFLAGS="$old_CFLAGS" fi fi if test "x$ORACLE_INCLUDE" = "x"; then AC_MSG_RESULT(no) AC_MSG_WARN([oracle headers not found. Use --with-oracle-include-dir=.]) fail="$fail oci.h" else sql_oracle_cflags="${sql_oracle_cflags} ${ORACLE_INCLUDE}" AC_MSG_RESULT(yes) if test "x$ORACLE_LIBS" = "x"; then AC_MSG_WARN([oracle libraries not found. Use --with-oracle-lib-dir=.]) fail="$fail libclntsh libnnz${oracle_version}" else sql_oracle_ldflags="${sql_oracle_ldflags} $ORACLE_LIBS" AC_MSG_RESULT(yes) fi fi targetname=modname else targetname= echo \*\*\* module modname is disabled. fi dnl Don't change this section. if test "x$fail" != x; then if test "x${enable_strict_dependencies}" = xyes; then AC_MSG_ERROR([set --without-]modname[ to disable it explicitly.]) else AC_MSG_WARN([silently not building ]modname[.]) AC_MSG_WARN([FAILURE: ]modname[ requires:$fail.]); targetname= fi fi AC_SUBST(sql_oracle_ldflags) AC_SUBST(sql_oracle_cflags) AC_SUBST(targetname) AC_OUTPUT(Makefile)

FreeRADIUS needs some library and header files to be integrated with Oracle, these files are shipped with Oracle Instant Client and Oracle Full Package both. You can tell FreeRADIUS where to find these files either by setting some system variables before executing the configure script or by passing some arguments to the configure script. We will stick to the second option here.

On my system, I had Oracle Instant Client (both Basic and SDK Packages) unzipped in /opt/instantclient/ so all of my references to Oracle are based on this path.

Now go back to FreeRADIUS’s untarred directory i.e. /usr/local/src/freeradius-server-2.1.10. Now you can run configure script from here by executing following command.

./configure --with-oracle-version=VER --with-oracle-include-dir=DIR --with-oracle-lib-dir=DIR

On my system this command takes the following form having instant client 11g on the path /opt/instantclient/.

./configure --with-oracle-version=11 --with-oracle-include-dir=/opt/instantclient/sdk/include/ --with-oracle-lib-dir=/opt/instantclient/

On a system with full Oracle 11g Installation, I had to execute the following command.

./configure --with-oracle-version=11 --with-oracle-include-dir=/u01/app/oracle/product/11.2.0/dbhome_1/rdbms/public/ --with-oracle-lib-dir=/u01/app/oracle/product/11.2.0/dbhome_1/lib/

Note: If you are using Instant Client or full Oracle 10g, you don’t have to replace the above ini file nor do you need to send the argument --with-oracle-version=VER to the configure script.

Rest of the commands for FreeRADIUS is quite common as below:

make

make install

I hope this post is helpful to those who are facing this problem.

Image Processing: Contrast and Brightness

2010-03-14T20:17:00.000+11:00

In Computer Graphics, RGB color model is often used to describe an image which means that the appearance of a pixel (often referred as Picture Element) is characterized by three primary bands i.e. Red, Green and Blue. Different kinds of effects with respect to the appearance of the image are created by manipulating the RGB values of pixels of the image. These effects include image blurring, rotation, edge detection, sharpening and different kinds of transformations etc. Image's contrast and brightness are two of those effects related to the appearance the image. There exist other models including HSV, HSI, YCbCr etc which can also be used to create these types of visual effects.
A pixel of a color image is represented by 24 bits (8 bits for each primary band which may assume any value between 0 to 255) in contrast to binary images, grey-scale images etc where the bits to represent the pixel may be less. However, another 8 bits are sometimes added (as is the case with Java) to represent the pixel which stands for alpha value representing transparency (i.e. opacity) of the pixel hence making a total of 32 bits to represent the pixel. There is plenty of information available on the web about different color models and different ways for their representation and manipulation. Here we want describe about an image's contrast and brightness.
The contrast of an image is defined by the distribution of the color values of the pixels of the image i.e. in a high contrast image; the color values of pixels exist over a wide distribution. On the other hand, a low contrast image, they exist in a narrow distribution. Hence color values are grouped together tightly over a short range of value for low contrast and are grouped loosely over a wide range of values for high contrast images. This grouping of color values also has an effect on the appearance of the image called Brightness.

In case of High Contrast Image, image appearance is much detailed while most of the details disappear in case of low contrast image. That's why often contrast enhancement is performed on images to make details visible which cannot be seen in low contrast. Coming back to the location of grouping of color values, the image may appear in some form grey color depending upon the location of grouping. That is if the location of the grouping of the values is (or close to) the maximum extreme (e.g. for RGB model, bands have a value of or near to 255), the image appears to be whitish while if location of grouping is (or close to) the minimum extreme (e.g. for RGB model, bands have a value of or near to 0), the image appears to be blackish. Therefore, this location of grouping of values make image look bright or dark.

Contrast of an image can be increased (or decreased) by increasing (or decreasing) the width of the color distribution. Brightness of an image can be increased by shifting the location of the grouping of values towards the maximum extreme value or can be decreased by shifting the location of grouping towards the minimum extreme value.

Histogram Analysis of an image is an analytic approach for studying images which is graphical representation of the color values occurring in the image, hence a representation of image contrast and brightness. The histogram shows the number of occurrences in an image of all possible values. A technique based on Histogram Analysis is called Histogram Equalization that is used controlling image contrast and brightness. Histogram Equalization can be performed in ways either Global Histogram Equalization (GHE) on the whole of image or Local Histogram Equalization in which case equalization is performed on a section or sections of an image. Generally, equalization is performed on grey-scale images, for color images equalization is performed on each primary band if color model is RGB. However, it is recommended to convert the image into some other color model before applying Histogram Equalization. For further details see following links:
Wikipedia Histogram Equalization
Ruye Wang Histogram Equalization
HIPR2 Histogram Equalization

Getting Started with BlackBerry J2ME Development

2009-07-09T05:20:00.000+10:00

Check out my article on java.net on blackberry application development with J2ME.

J2ME: Browsing File System on Mobile Device using FileConnection APIs

2009-02-21T12:30:00.000+11:00

There can be application where we need to browse through different files on our mobile phone, File Viewer, File Uploader are few to name. Here I’m going to provide a complete program that helps you browse through your file on mobile device and therefore helps you develop more sophisticated applications using it. Its based on J2ME FileConnection APIs (JSR 75) which is a optional package within MIDP. Though the midlet code provided is quite simple and straightforward and needn’t much explanation, however, please consult JavaDoc for FileConnection API for details which is available here. The midlet allows the user to browse through file system on the Java Enabled devices.
There are some other ways to open file connection but they open connection to specific folders e.g. System.getProperty("fileconn.dir.photos") which retruns the path to ‘photos’ folder to which connection can be opened. Moreover this method might not be supported by the some of the devices. In my case, I tried this approach for SE K800i and Nokia 6630 where it worked but not for Motorola V3. In that case, the approach outlined by the code snippet here is more suitable.

Note: FileConnection APIs (JSR 75) being an optional package of MIDP might not be available on the all the devices. So you’ll have to check if the particular device has support for it using system property System.getProperty("microedition.io.file.FileConnection.version").

/**
*
* @author Atifs
*/
import javax.microedition.lcdui.*;
import javax.microedition.midlet.MIDlet;

public class BrowserLauncher extends MIDlet implements CommandListener {

private Display display;
private final Command CMD_EXIT = new Command("Exit", Command.EXIT, 1);
FileSysBrowser fsb;

public BrowserLauncher() {
display = Display.getDisplay(this);
}

protected void startApp() {
fsb = new FileSysBrowser(this);
fsb.showFiles();
}

public Display getDisplay() {
return display;
}

protected void destroyApp(boolean unconditional) {
}

protected void exitApp() {
destroyApp(false);
notifyDestroyed();
}

protected void pauseApp() {
}

public void commandAction(Command c, Displayable d) {
if (c == CMD_EXIT) {
exitApp();
}
}
}

/**
*
* @author Atifs
*/
import java.io.IOException;
import javax.microedition.lcdui.*;
import javax.microedition.io.*;
import javax.microedition.io.file.*;
import java.util.*;

public class FileSysBrowser extends List implements CommandListener {
// 'DATHHEAD' holds the root, please see javadoc for details
// This may vary for different devices.
String DATHHEAD = "file:///";
private Command CMD_SUBMIT = new Command("Open", Command.ITEM, 0);
private Command CMD_EXIT = new Command("Exit", Command.EXIT, 1);
private Command CMD_BACK = new Command("Back", Command.BACK, 1);
private BrowserLauncher fl;
private String path = null;
private String selectedItem = null;
FileConnection srcconn = null;

public FileSysBrowser(BrowserLauncher fl) {
super("File List", List.IMPLICIT);
addCommand(CMD_SUBMIT);
addCommand(CMD_EXIT);
addCommand(CMD_BACK);
setCommandListener(this);
this.fl = fl;
}

protected void showFiles() {
if (path == null) {
Enumeration e = FileSystemRegistry.listRoots();
path = DATHHEAD;
setTitle(path);
while (e.hasMoreElements()) {
String root = (String) e.nextElement();
// you can pass an 'image' object instead of 'null' to
// append method
append(root, null);
}
fl.getDisplay().setCurrent(this);
} else {
if (selectedItem != null) {
setTitle(path + selectedItem);
} else {
setTitle(path);
}
try {
// works when users opens a directory
if (selectedItem != null) {
srcconn = (FileConnection) Connector.open(path + selectedItem, Connector.READ);
} else // works when presses 'Back' to go one level above/up
{
srcconn = (FileConnection) Connector.open(path, Connector.READ);
}
// Check if the selected item is a directory
if (srcconn.isDirectory()) {
if (selectedItem != null) {
path = path + selectedItem;
selectedItem = null;
}
Enumeration files = srcconn.list();
while (files.hasMoreElements()) {
String file = (String) files.nextElement();
append(file, null);
}
//
fl.getDisplay().setCurrent(this);
try {
if (srcconn != null) {
srcconn.close();
srcconn = null;
}
} catch (IOException ex) {
ex.printStackTrace();
}
}//if (srcconn.isDirectory())
else {
// 'else' is unnecessary here, i just out here to signify the
// fact that you handle the selected item if it is a 'file'
// instead of a directory. For example you can display a file
}
} catch (Exception e) {
e.printStackTrace();
}
}
}

protected String getDirectoryName() {
int select = getSelectedIndex();
return getString(select);
}

public void commandAction(Command c, Displayable d) {
if (c == CMD_SUBMIT) {
selectedItem = getDirectoryName();
deleteAll();
showFiles();
}
if (c == CMD_EXIT) {
fl.exitApp();
}
if (c == CMD_BACK) {
if (!path.equals(DATHHEAD)) {
int slashIndex = path.lastIndexOf('/', path.length() - 2);
path = path.substring(0, slashIndex + 1);
if (path.equals(DATHHEAD)) {
path = null;
}
deleteAll();
showFiles();
}
}
}
}

Disclaimer: The code provided has been written for exploratory purposes, not following the official guideline of Software Engineering and therefore has not been tested thoroughly. However, you are FREE to use the code for any purpose and be aware to test it before using for production uses!!!

J2ME: Thumbnails Extraction of JPEG (Exif) Images made with Mobile Phone Camera

2009-02-04T23:09:00.000+11:00

....continuation

Following are the code snippets for extracting the thumbnail from JPEG image/picture created from mobile phone camera. I have put comments inside the code where necessary, I guess they should be enough to explain the code, however, I would strongly recommend to study this link for understanding the Exif format. This will really help you understand the code. Please the read the comments carefully!!

Disclaimer: The code provided has been written for exploratory purposes (that’s why there are so many console output and debug statements), not following the official guideline of Software Engineering and therefore has not been tested thoroughly but it does provide a good foundation and good starting point if you intend to work on the topic. However, you are FREE to use the code for any purpose and be aware to test it before using for production uses!!!

To start with you need an object of ‘InputStream’ opened to the JPEG file. (We will discuss at some other time how to open a connection to a file). Pass this object to the extractThumbnail(). Rest of stuff is pretty much clear from the comments so I leave you here with it!!!

/*
This method is the starting point, it takes an 'InputStream' object as an
argument to the JPEG file
*/
public int extractThumbnail(InputStream in) throws Exception {
int current;
if (in == null) {
return 0;
}
current = read16Bytes(in);
if (current != 0xFFD8) { //SOI
error("Not a JPEG file");
return 0;
}
current = read16Bytes(in);

// Reading until 'APP' marker is reached
while (!(current == 0xFFE0 || current == 0xFFE1 || current == 0xFFE2 ||
current == 0xFFE3 || current == 0xFFE4 || current == 0xFFE5 ||
current == 0xFFE6 || current == 0xFFE7 || current == 0xFFE8 ||
current == 0xFFE9 || current == 0xFFEA || current == 0xFFEB ||
current == 0xFFEC || current == 0xFFED || current == 0xFFEE ||
current == 0xFFEF)) {
current = read16Bytes(in);
if (current == 0xFFD9) { // End of Image
error("ERROR: File Malformed");
return 0;
}
}
return readAppMarker(in);
}

// Utility Method to throw exceptions
private void error(String message) throws Exception {
throw new Exception(message);
}

// Utility Method to read 2 bytes from the file*/
private final int read16Bytes(InputStream in) throws Exception {
int temp;
try {
temp = in.read();
temp <<= 8;
return temp | in.read();
} catch (IOException e) {
error("read16Bytes() read error: " + e.toString());
return -1;
}
}

// Utility Method to read 1 bytes from the file
private final int read8Bytes(InputStream in) throws Exception {
try {
return in.read();
} catch (IOException e) {
error("read8Bytes() read error: " + e.toString());
return -1;
}
}

// Method that reads the App Marker from the file to the point where
// thumbnail exists...
private int readAppMarker(InputStream in) throws Exception {
boolean debug = false;
// 'Lp' holds the lenght of the marker
int Lp;
// 'content' contains the bytes read from the file
int content = -1;
//'count' for the bytes read from the file
int count = 0;
int offset = 0;
int entries = 0;
// Readin the lenght for marker
Lp = read16Bytes(in);
count += 2;
System.out.println("Length of App1 marker: " + Lp);
if (debug)
System.out.println("Reading App1 identifier\nIdentifier: ");

// Reading the identifier i.e.6 bytes
int iden = 0;
String id = null;
content = read8Bytes(in); //1st byte read of the identifier
count++;
id = Integer.toHexString(content); // This byte should be '45' for Exif or '4a' for JFIF
if (!id.equals("45")) {
while (iden < 5) { // Reading the remaining 5 bytes which are 78, 69, 66, 00, 00
content = read8Bytes(in);
count++; // Incrementing each time a byte is read
iden++;
if (debug)
System.out.print(Integer.toHexString(content));
}
} else {
// Not an exif image
return 0;
}

/* If the image is found to be ''Exif', remaining
* bytes will be following 'TIFF' image format
*/

if (debug) {
System.out.println("\nBytes read for identifier: " + (count - 2));
System.out.println("\n\nReading TIFF Header");
}
// 'tiffCounter' contains the bytes read from the file
// Though i could have used 'count'.....
int tiffCounter = 0;
// 'intelAlign' used as a flag indiacte byte alignment
// either intel (little endian) or
// non-intel(motorola-big endian)
boolean intelAlign = false;

// Reading first 8 bytes for TIFF header
while (tiffCounter < 8) {
if (tiffCounter == 0) {
content = read16Bytes(in);
// count+=2;
tiffCounter += 2;
if (Integer.toHexString(content).equals("4949")) {
intelAlign = true;
}
if (debug) {
System.out.println("Byte Align: " + Integer.toHexString(content));
}
} else {
if (tiffCounter == 2) {
content = read16Bytes(in);
// count += 2;
tiffCounter += 2;
if (debug) {
System.out.println("Tag mark: " + Integer.toHexString(content));
}
} else if (tiffCounter == 4) {
// Reading offset for IFD0 which conatins
// offset for IFD1 which in turn conatins the
// thumbnail
content = read16Bytes(in);
content <<= 16;
content |= read16Bytes(in);
if (intelAlign) {
content = convertIntegerToBigEndian(content);
}
offset = content;
// count += 4;
tiffCounter += 4;
if (debug) {
System.out.println("IFD0 Offset: " + Integer.toHexString(offset));
}
}
}
}
if (debug) {
System.out.println("Bytes read for header: " + tiffCounter);
System.out.println("\n\nBytes to be read for IFD0: " + (offset - tiffCounter));
}

// Reading bytes until reached IFD0
while (offset - tiffCounter != 0) {
content = read8Bytes(in);
tiffCounter++;
if (debug) {
System.out.println("Bytes to be read for IFD0: " + (offset - tiffCounter));
}

}

// Reading IFD0 .......to get the offset for FD01 which
// contains Thumbnail
content = 0;
// reading the number of entries in IFD0
content = read16Bytes(in);
if (intelAlign) {
content = convertShortToBigEndian(content);
}

tiffCounter += 2;
entries = content;
int bytesReadSoFar = tiffCounter;
if (debug) {
System.out.println("\n\nNo. of entries in IFD0: " + entries);
}

// Reading all the entries in IFD0
// 12 bytes for each entry
// The loop is bit elaborated although i could have just skim
// all those byte for all the entries
// I was just exploring it:)
for (int i = 0; i < entries; i++) {

int entryCounter = tiffCounter;
if (debug) {
System.out.println("\n\nScaning entry in IFD0 no. : " + (i + 1));
}
content = read16Bytes(in);
tiffCounter += 2;
if (debug) {
System.out.println("Tag no. : " + content);
}

content = read16Bytes(in);
tiffCounter += 2;
if (debug) {
System.out.println("Data Format : " + content);
}

content = read16Bytes(in);
content |= read16Bytes(in);
tiffCounter += 4;
if (debug) {
System.out.println("no. of Components : " + content);
}

content = read16Bytes(in);
content |= read16Bytes(in);
tiffCounter += 4;
if (debug) {
System.out.println("Either data value or offset to data value : " + content);
System.out.println("Bytes read for entry : " + (tiffCounter - entryCounter));
}
}

if (debug) {
System.out.println("Bytes read for IFD0 : " + (tiffCounter - bytesReadSoFar));
}

// Reading offset for IFD1
content = 0;
content = read16Bytes(in);
content <<= 16;
content |= read16Bytes(in);
if (intelAlign) {
content = convertIntegerToBigEndian(content);
}
tiffCounter += 4;
offset = content;
if (debug) {
System.out.println("\n\nBytes to be read for IFD1 " + (offset - tiffCounter));
}

// Reading bytes until reached IFD1
while (offset - tiffCounter != 0) {
content = read8Bytes(in);
tiffCounter++;
if (debug) {
System.out.println("Bytes to be read for IFD1: " + (offset - tiffCounter));
}
}

// Reading number of entries for IFD1
content = read16Bytes(in);
if (intelAlign) {
content = convertShortToBigEndian(content);
}
tiffCounter += 2;
entries = content;
bytesReadSoFar = tiffCounter;
if (debug) {
System.out.println("\n\nNo. of entries in IFD1: " + entries);
}

// 'data' array contains the info about the thumbnail
int data[] = new int[3];
// Reading the entried for IFD1
// 12 bytes for each entry
for (int i = 0; i < entries; i++) {
int desiredTag = -1;
int entryCounter = tiffCounter;
if (debug) {
System.out.println("\n\nScaning entry in IFD1 no. : " + (i + 1));
}
content = read16Bytes(in);
content = convertShortToBigEndian(content);
tiffCounter += 2;
// tag '103' containd about the format of thumbnail i.e. Jpeg compression
// or TIFF format howeve, we are interested in JPEg compression
// JPEG compression has been standardised by DCS so we expect it to
// JPEG compression for mobile phone cameras
if (Integer.toHexString(content).equals("103")) {
desiredTag = 0;
}
// if JPEG compression tag '201' contains the offset to the thumbnail image
if (Integer.toHexString(content).equals("201")) {
desiredTag = 1;
}
// if JPEG compression tag '202' contains the size of the thumbnail
if (Integer.toHexString(content).equals("202")) {
desiredTag = 2;
}

if (debug) {
System.out.println("Tag no. : " + Integer.toHexString(content));
}

content = read16Bytes(in);
content = convertShortToBigEndian(content);
tiffCounter += 2;
if (debug) {
System.out.println("Data Format : " + content);
System.out.println("Data Format Hex : " + Integer.toHexString(content));
}

content = read16Bytes(in);
content <<= 16;
content |= read16Bytes(in);
content = convertIntegerToBigEndian(content);
tiffCounter += 4;
if (debug) {
System.out.println("no. of Components : " + content);
System.out.println("no. of Components : " + Integer.toHexString(content));
}

content = read16Bytes(in);
content <<= 16;
content |= read16Bytes(in);
content = convertIntegerToBigEndian(content);
tiffCounter += 4;
if (desiredTag != -1) {
data[desiredTag] = content;
}
if (debug) {
System.out.println("Either data value or offset to data value : " + content);
System.out.println("Either data value or offset to data value : " + Integer.toHexString(content));
System.out.println("Bytes read for entry : " + (tiffCounter - entryCounter));
}
}
if (debug) {
System.out.println("Bytes read for IFD1 : " + (tiffCounter - bytesReadSoFar));
}

/*
'data[0]' conatins the thumbnail format
'data[1]' contains the offset to the thumbnail
'data[2]' contains the size of the thumbnail in bytes (which we might not need as we'll pass
to a JPEG Decoder which will take care of it...in fact the length is already there in JPEG
thumbnail image!!!)
*/

// we only proceed only if the thumbnail is JPED compressed
// for which data[0] = 6
if (data[0] != 6) {
error("Not a JPEG file");
return 0;
} else {
// reading bytes until reached the thumbnail
while (data[1] - tiffCounter != 0) {
content = read8Bytes(in);
tiffCounter++;
if (debug) {
System.out.println("Bytes to be read for IFD1: " + (data[1] - tiffCounter));
}
}
}
if (debug) {
System.out.println("Total Length for the marker: " + Lp);
System.out.println("Total bytes read so far: " + (tiffCounter + count));
System.out.println("Bytes left for this marker");
}

/*
At this point, we need a JPEG decoder which starts reading the file
* from this point in the file, that decodes the thumbnail. Remember
* thumbnail is reduced version of actual image but still a JPEG image
* if JPEG Compressed!!
*/
return 1;

}

// Utility Method to convert Short (2bytes) from Little Endian to
// Big Endian
int convertShortToBigEndian(int i) {
return ((i >> 8) & 0xff) + ((i << 8) & 0xff00);
}

// Utility Method to convert Integer (4bytes) from Little Endian to
// Big Endian
int convertIntegerToBigEndian(int i) {
return ((i & 0xff) << 24) + ((i & 0xff00) << 8) + ((i & 0xff0000) >> 8) + ((i >> 24) & 0xff);
}

After the execution of extractThumbnail() method you are at the point where thumbnail exists in the file, it should be noted that thumbnail is scaled down version of the actual image, but some of the markers are missing here. You need a JPEG decoder that decodes the thumbnail for display; there is one free JPEG Decoder available here. It also instructs how to incorporate this decoder in your code.

Well, you might be wondering why we are extracting thumbnail if we still need a JPEG decoder!!! Why don’t we just decode the actual image and produced a reduced version of the image using above mentioned decoder?? The answer is, as I experienced it, it takes very long to decode the full image and producing a thumbnail, the situation is even worst while dealing with image of very high dimension e.g. 2048x1680, which is not good from your application point of view. So it’s a good idea to use the already embedded thumbnail which is far quicker to decode for the decoder, moreover, we should use this if already there because that’s the whole purpose of it!!!!!

J2ME: Thumbnails Extraction of JPEG (Exif) Images made with Mobile Phone Camera

2009-01-21T23:36:00.000+11:00

Mobile Phones Cameras, being digital, make pictures/images in JPEG format particularly ‘Exif’ flavor of JPEG format. Exif has been decided as a standard for Digital Cameras. We are not going to discuss the process involved in the creation of JPEG images i.e. the encoding process of a raw image into compressed JPEG image which essentially involves three steps in order of Discrete Cosine Transform(DCT), Quantization and Entropy (Huffman) Encoding. Here we are concerned about how data is organized that is the ‘FORMAT’ once a JPEG image has been created. However we do need a JPEG decoder for thumbnail extraction, there is one freely available. First we talk a little about JPEG format and its Exif flavor.
Few Words on JPEG format
A JPEG (jpg) file is organized in order of markers along with their contents. Each marker itself takes 2 bytes. The very first marker (0xFFD8) stands for Start of Image (SOI). This declares that this is a JPEG file. The second marker is (APPn) that depends upon the application using JPEG hence the marker contains an identifier in its contents indicating the application. The marker have any value from APP0 (0xFFE0) to APP15 (0xFFEF) both inclusive.
In JPEG format, immediate 2 bytes after each marker contain the length of the marker’s contents including the length field itself, so is the case with APPn marker. APP0 (0xFFE0) belongs to ‘JFIF’ marker while APP1 (0xFFE1) to ‘Exif’ marker (We are interested in APPn marker particularly APP1 (0xFFE1) marker for extracting thumbnail which is already embedded in the file within this marker). In APPn, after length field, following bytes contain the ASCII code equivalent of the identifier name (5 bytes for ‘JFIF’ and 6bytes for ‘Exif’). Please see the NOTE below. The Exif identifier is 45, 78, 69, 66, 00, 00 (6 bytes). From there on, Exif format is same as TIFF image format is. More detail on Exif (and its embedded TIFF) please see here. After reading a specific number of offset bytes when thumbnail offset is reached, it could be in one of three formats JPEG compressed (most commonly used), RGB TIFF or YCbCr TIFF (The number of offset bytes depends upon ‘byte align’ discussed below). If JPEG compressed, it is just like another JPEG image of a cut down scale which is decoded for display. (I have tried on Sony Ericson K800i and Nokia 6630, both of them had the thumbnail in JPEG Compressed format hence our discussion pertains to this only). Another important thing about TIFF header (8 bytes), embedded in Exif format, is that its first 2 byte informs you about the byte align of TIFF data to be followed that is either little endian (used by Intel) or big endian (used by Motorola). So you have got to look for this thing to calculate the offset while reading Exif file in general and TIFF file in particular. JPEG generally uses big endian however Exif allows both of them. Moreover, most of the digital cameras using Exif format follow little endian. Also remember that all the offset in TIFF are calculated from the first byte of the TIFF header.
NOTE
In case of JFIF, last byte out of 5 contains zero while last two bytes in case of ‘Exif’ contain zero, for example, 4A, 46, 49, 46, 00 (5 bytes)for JFIF. The JFIF v1.02 and above have JFIF extension according to which APP0 has extension part which again starts with application marker hence making two APP markers. The thumbnail may be located either under first marker or under second marker For more detail on JFIF , please see here.
...continued

JDOM vs DOM4J

2009-01-15T08:51:00.000+11:00

Bear in mind, JDOM and DOM4J are java API for XML processing. JDOM is open source pure Java API for XML processing. JDOM is not an extension or wrapper over W3C DOM model. JDOM is an object model in java that stands for XML document that is the same model as of DOM. JDOM does use SAX API with XML parser (Xerces or Crimson or any other) at the back end to establish the in-memory tree based model.
Note: JDOM uses the defaults parser of JAXP as it calls the parser through JAXP, however, it can be changed to any other parser (Please see the previous post).
You can find this description on the following this link. Like DOM and unlike to SAX, JDOM provides random access to contents of the document hence allowing to update it. JDOM is compatible for conversion to DOM Model or SAX Events and vice versa. Please see this. Unlike DOM, JDOM doesn’t work through interfaces rather it has concrete classes for different components of the model i.e. node, element, attributes etc.
DOM4J, on the other hand like DOM, has done it other way round than JDOM. DOM4J is also XML Processing APIs and follow the same tress based object model for XML document. DOM4J works through interfaces rather than concrete classes. That’s the major difference between DOM4J and JDOM. DOM4J makes navigating along the tree easier than JDOM due to this difference. DOM4J also provides some support for XPath (which, unfortunately, I didn’t get a chance to explore).
The purpose of this and the last three posts is to have a clear understanding of what is what in the area of XML Parsing. I haven’t provided any code here because there is huge coding stuff available on the web for this purpose. I’ll get back to each of them with more detail some other time (fingers crossed!!).

JAXP: Another Player in Java-XML Game

2009-01-03T00:35:00.000+11:00

JAXP stands for ‘Java API for XML Processing’. Please, don’t confuse it with other APIs like SAX, DOM, JDOM etc. JAXP is a set of APIs for XML processing making a rather abstract layer in the overall scheme of XML processing. SAX, DOM and JDOM are parsing APIs only, used in parsing of XML document, while JAXP has a broader spectrum of its application, it lets you parse, validate and transform XML documents, here we are concerned with the parsing feature of it however. Remember, JAXP is not parsing APIs nor is it a new way of XML handling in java. As a matter of fact, JAXP provides a convenient way of XML processing. As said before, JAXP exists at the abstract layer; it uses a parser behind the scenes for parsing purposes. JAXP is a bundle of some APIs and a parser. Previously, JAXP was bundled with Crimson parser as the default one; however, JAXP provides the facility to change to some other parser without recompiling the application which has several benefits. (Details can be found on the following link.) These parsers in turn implement SAX, DOM or some other parsing APIs. We are not going into much detail here; please refer to All About JAXP for detail on JAXP along with example code too, this tutorial on JAXP provides quite a comprehensive detail on JAXP highlighting its parsing and validation feature.

When to use SAX and DOM?

2008-12-22T19:15:00.000+11:00

This question doesn’t have any straight or fix answer because it really depends upon the situation. Different parameters (memory, speed etc) have to be taken into consideration to make this decision. We go through some of points that will be helpful in making the decision which API should be used in which situation.

DOM is memory intensive while SAX is not, because DOM creates a Document object in memory using a tree data structure while SAX does not create any such default object in memory rather uses custom object which makes SAX less memory intense.
DOM is slower than SAX for the same reason as of memory.
DOM provides a random access to the XML contents of the document having them in memory while SAX provides sequential access to the contents as it has access to one part of the document at a time.
DOM has an edge over SAX that it provides ability to modify the document which is not the case with SAX. This implies that if client application is concerned about the location of different elements within the document (may be to modify it) then DOM is the option, however, if it’s concerned with individual elements then SAX is the right option.
From a parser’s point of view, there is nothing much to do for a parser for SAX. There are few interfaces to be implemented for SAX for parser while this is not the case with DOM. Most of the parsers support both of these models.

Based on these points, you can have some idea which model is the most suitable to your needs!!

XML Parsing

2008-12-16T11:06:00.000+11:00

With respect to XML document parsing particularly with Java, things have been quite puzzling, at least for me! Jargons like DOM, SAX, JDOM, dom4j etc made things quite confusing! I am going to put up my finding on the issue, from different sources, to share with others, hope it helps. Please correct me or share with me if I am wrong on something or if something is missing!! Let us talk a little about DOM and SAX first then JDOM and dom4j.
Generally an XML document can be parsed in one of two ways either via Document Object Model (DOM) that’s been specified by W3C or via Simple API for XML (SAX).
By looking at an XML document (at least a well-formed one), it’s quite simple to say that it has a tree like structure e.g. root node(root element), child nodes, sub-child nodes etc. So we can infer that a tree structure will help us access the contents within the document, tree data structure for instance!!! This is the approach around which DOM APIs are based on. On the other hand, XML document contents can also be accessed by the reading the document sequentially, very much like humans do while reading a book, that’s where SAX APIs have originated from!! !
Document Object Model, or DOM, is a platform and language-independent API from W3C for accessing and modifying XML documents. A DOM based parser reads the whole XML document in one go and keeps it in memory in tree-like structure where each node of the tree stands for an element of the document. Once in memory, parser provides methods to access the contents. The XML document is read line by line and as element(s) are encountered, the corresponding tree is built on. DOM enables navigation along the built tree in any direction providing random access to any node.
Simple API for XML, or SAX, is API collection of Callbacks for sequential parsing of the XML document. SAX based parsers are event-driven parser, these parsers parse XML document sequentially without storing the structure into the memory that makes SAX memory efficient as compared to DOM. The events issued by the parser are: start and end of the document, start and end of each tag, comments and processing instructions. SAX based parsers, being sequential, provides access to only one bit of the document at a time.

Remember: SAX and DOM are just APIs for parsing XML document, (so are JDOM and Dom4j as we’ll discuss them later) and are not parsers!!! You do need DOM, SAX or any other parsing API to parse XML Document. A parser can be based on either of the APIs.

SAX and DOM APIs have evolved as specifications that describe how XML parsers can pass contents of an XML documents to client applications through different interfaces which are implemented by XML parsers. Though first developed for Java, APIs were later developed for other major programming languages. There are number of parsers available Apache Xerces, Sun Crimson, Oracle’s XML Parser, MSXML etc. Apache Xerces and Crimson have been included in the JDK 1.5 and later.

SAX and DOM APIs are different with respect to their structure as explained above that is SAX is an event based collection of Callbacks while DOM is in-memory tress structure. Thus DOM uses tree data structure while does SAX doesn’t use any until done so manually which essentially means that a parse tree can also be constructed with event based API.

Next time we'll discuss a comparison of DOM and SAX