Hunter

Java useful command

2013-12-08T21:26:00.003-08:00

-To run java program in debug mode
pass -Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=<port_NO> as JVM arguments

Windows useful commands and default directory structure common software.

2013-12-05T23:26:00.006-08:00

To start service in window OS from command line.

net start/stop/pause/continue <<service_name>>

To start Clear case service in Windows.

net start albd.

Most used Commands and Standered file directory path in Linux

2013-11-26T23:28:00.001-08:00

To modify host name and IP address in Linux

/etc/hosts file

create bond( /etc/sysconfig/network-scripts/ifcfg-bond0)

DEVICE=bond0

ONBOOT=yes

USERCTL=no

BOOTPROTO=none

BONDING_OPTS="primary=eth0"

IPADDR=xx.xx.xxx.xx

NETMASK=xxx.xxx.xxx.x

GATEWAY=xx.xx.xxx.x

BROADCAST=xx.xx.xxx.xxx

configure net work(/etc/sysconfig/network-scripts/ifcfg-eth0

DEVICE=eth0

ONBOOT=yes

USERCTL=no

BOOTPROTO=none

SLAVE=yes

MASTER=bond0

change the domine name

/etc/sysconfig/network(hostname)

/etc/hosts

service network restart

To delete user in Linux

userdel -r userName(-r used to delete along with home directory)

12 Little Known Laws of Karma (That Will Change Your Life)

2013-09-16T23:30:00.002-07:00

What is Karma? Karma is the Sanskrit word for action. It is equivalent to Newton's law of ‘every action must have a reaction’. When we think, speak or act we initiate a force that will react accordingly. This returning force maybe modified, changed or suspended, but most people will not be able eradicate it.

This law of cause and effect is not punishment, but is wholly for the sake of education or learning. A person may not escape the consequences of his actions, but he will suffer only if he himself has made the conditions ripe for his suffering. Ignorance of the law is no excuse whether the laws are man-made or universal. To stop being afraid and to start being empowered in the worlds of karma and reincarnation, here is what you need to know about karmic laws.

THE GREAT LAW

- "As you sow, so shall you reap". This is also known as the "Law of Cause and Effect".
- Whatever we put out in the Universe is what comes back to us.
- If what we want is Happiness, Peace, Love, Friendship... Then we should BE Happy, Peaceful, Loving and a True Friend.

THE LAW OF CREATION

- Life doesn't just HAPPEN, it requires our participation.
- We are one with the Universe, both inside and out.
- Whatever surrounds us gives us clues to our inner state.
- BE yourself, and surround yourself with what you want to have present in your Life.

THE LAW OF HUMILITY

- What you refuse to accept, will continue for you.
- If what we see is an enemy, or someone with a character trait that we find to be negative, then we ourselves are not focused on a higher level of existence.

THE LAW OF GROWTH

- "Wherever you go, there you are".
- For us to GROW in Spirit, it is we who must change - and not the people, places or things around us. - The only given we have in our lives is OURSELVES and that is the only factor we have control over. - When we change who and what we are within our heart our life follows suit and changes too.

THE LAW OF RESPONSIBILITY

- Whenever there is something wrong in my life, there is something wrong in me.
- We mirror what surrounds us - and what surrounds us mirrors us; this is a Universal Truth.
- We must take responsibility what is in our life.

THE LAW OF CONNECTION

- Even if something we do seems inconsequential, it is very important that it gets done as everything in the Universe is connected.
- Each step leads to the next step, and so forth and so on.
- Someone must do the initial work to get a job done.
- Neither the first step nor the last are of greater significance,
- As they were both needed to accomplish the task.
- Past-Present-Future they are all connected...

THE LAW OF FOCUS

- You can not think of two things at the same time.
- When our focus is on Spiritual Values, it is impossible for us to have lower thoughts such as greed or anger.

THE LAW OF GIVING AND HOSPITALITY

- If you believe something to be true,then sometime in your life you will be called upon to demonstrate that particular truth.
- Here is where we put what we CLAIM that we have learned, into actual PRACTICE.

THE LAW OF HERE AND NOW

- Looking backward to examine what was, prevents us from being totally in the HERE AND NOW.
- Old thoughts, old patterns of behavior, old dreams... - Prevent us from having new ones.

THE LAW OF CHANGE

- History repeats itself until we learn the lessons that we need to change our path.

THE LAW OF PATIENCE AND REWARD

- All Rewards require initial toil.
- Rewards of lasting value require patient and persistent toil. -
True joy follows doing what we're suppose to be doing, and waiting for the reward to come in on its own time.

THE LAW OF SIGNIFICANCE AND INSPIRATION

- You get back from something whatever YOU have put into it.
- The true value of something is a direct result of the energy and intent that is put into it.
- Every personal contribution is also a contribution to the Whole.
- Lack luster contributions have no impact on the Whole, nor do they work to diminish it.
- Loving contributions bring life to, and inspire, the Whole.

Sources: Bliss Returned and Raven Emrys

Java technology, IBM style: Garbage collection policies, Part 1

2013-02-25T00:32:00.000-08:00

Differing policies provide flexibility

Summary: One of the great benefits of the Java™ platform is that it takes care of much of the work of garbage collection for you, but there are occasions when you still want to tweak the way garbage collection takes place. With the latest Java technology implementation from IBM®, you can choose among several garbage collection policies to help you get the most out of your application. In this second article in the series Java technology, IBM style, Java developer Mattias Persson explores the available options and details the situations in which each might be appropriate.

Garbage collection (GC) in the IBM Developer Kit for the Java 5.0 Platform (IBM SDK) can be configured using four different policies. This article, the first of two parts on GC, introduces the different garbage collection policies and discusses their general characteristics. You should have a basic understanding of garbage collection in the Java platform before you begin. Part 2 presents a quantitative approach to choosing a policy, along with with some examples.

About the series

The Java technology, IBM style series takes a look at the latest releases of the IBM implementations of the Java platform. You'll learn how IBM has implemented some of the advances built into version 5.0 of the Java platform, and you'll find out how to use some of the value-added features built into new releases from IBM.

Contact the authors individually with comments or questions about their articles. To comment on the series as a whole, you may contact series lead Chris Bailey. For more on the concepts discussed here and links where you can download the latest IBM releases, see Resources.

Why different GC policies?

The availability of differing GC policies gives you increased capabilities. There are many different algorithms available for GC, each of which has advantages and drawbacks, depending on the type of workload. (If you are unfamiliar with the general topic of GC algorithms, see Resources for further reading.) In the IBM SDK 5.0, you can configure the garbage collector with one of four policies, each of which uses its own algorithm. The default policy is sufficient for most applications. If you do not have any particular performance requirements for your application, then the content of this article (and the next) may not be interesting to you; you can run the IBM SDK 5.0 without changing the GC policy. However, if your application requires optimal performance or if you are generally concerned with the length of GC pauses, then read on. You will see that the latest release from IBM has even more choice than its predecessors.

So why doesn't the IBM implementation of the Java runtime make the choice automatically for you? This isn't always possible. It's difficult for the runtime to understand your needs. In some situations, you may want to run the application with high throughput; in others, you might prefer to reduce the pause times.

Table 1 lists the available policies and explains when you should use each one. The following sections describe the characteristics of each policy in more detail.

Table 1. GC policies in the IBM SDK 5.0

Policy	Option	Description
Optimize for throughput	`-Xgcpolicy:optthruput`(optional)	The default policy. It is typically used for applications where raw throughput is more important than short GC pauses. The application is stopped each time that garbage is collected.
Optimize for pause time	`-Xgcpolicy:optavgpause`	Trades high throughput for shorter GC pauses by performing some of the garbage collection concurrently. The application is paused for shorter periods.
Generational concurrent	`-Xgcpolicy:gencon`	Handles short-lived objects differently than objects that are long-lived. Applications that have many short-lived objects can see shorter pause times with this policy while still producing good throughput.
Subpooling	`-Xgcpolicy:subpool`	Uses an algorithm similar to the default policy's but employs an allocation strategy that is more suitable for multiprocessor machines. We recommend this policy for SMP machines with 16 or more processors. This policy is only available on IBM pSeries® and zSeries® platforms. Applications that need to scale on large machines can benefit from this policy.

Some terms defined

Throughput is the amount of data processed by an application. Throughput must be measured with an application-specific metric.

Pause time is the duration of time in which the garbage collector has paused all application threads to collect the heap.

In this article, I use the abbreviations from the command-line options detailed in Table 1 to denote these policies: optthruput for Optimize for throughput, optavgpause for Optimize for pause time,gencon for Generational concurrent, and subpool for Subpooling.

When should I consider a non-default GC policy?

I advise that you always start with the default GC policy. Before you move away from the default, you need to understand the circumstances under which you should explore an alternative policy. Table 2 presents a set of reasons that could apply:

Table 2. Reasons for switching to an alternative GC policy

Switch to	Reasons
`optavgpause`	My application cannot tolerate the length of the GC pauses. A degradation in performance is acceptable as long as the GC pause time is reduced. I am running on a 64-bit platform and use a very large heap -- more than 3 or 4GB. My application is a GUI application and I'm concerned about the user response times.
`gencon`	My application allocates many short-lived objects. The heap space is fragmented. My application is transaction-based (that is, objects in the transaction don't survive beyond the transaction commit).
`subpool`	I have scalability problems on large multiprocessor machine.

Let me stress that the reasons mentioned in Table 2 are not sufficient to conclude that the alternative policies will perform better; they are simply hints. In all cases, you should run the application and measure throughput and/or response times in combination with GC pause times. The next part of this series shows examples of this sort of testing.

The remaining sections of this article describe in more detail the differences between the GC policies.

optthruput

Signs of heap fragmentation

The most common sign of heap fragmentation is the presence of premature allocation failures. This is seen in verbose garbage collection (-Xverbose:gc) as GC being triggered even though there is free space on the heap. Another sign is the presence of small thread allocation caches (see the subsection "Heap lock and thread allocation caches"). You can use -Xtgc:freelist to determine the average size. Both the options are explained in the IBM SDK 5 Diagnostics Guide available in Resources.

optthruput is the default policy. It is a tracing collector, referred to as a mark-sweep-compact collector. The mark and the sweep phases always run during GC, but compaction only happens under certain circumstances. The mark phase finds and marks all live objects. The sweep phase removes all unmarked objects. A third and optional step is compaction. There are various circumstances under which compaction might occur; the most common one is the system's inability to reclaim enough free space.

Fragmentation occurs when objects are allocated and freed so often that only small chunks of free space are left in the heap. There may be a large amount of free space overall in the heap, but the contiguous regions are small, causing allocation failures.Compaction moves all the objects down to the beginning of the heap and aligns them so that no space exists between them. This removes fragmentation from the heap but is an expensive task, so it is only performed if necessary.

Figure 1 shows an outline of the heap layout after the different phases: mark, sweep, and compaction. The dark regions represent objects and the bright regions are free space.

Mark and sweep

The mark phase traverses all objects that can be referenced from thread stacks, statics, interned strings, and JNI references. As part of this process, we create a mark bit vector that defines the beginning of all live objects.

The sweep phase uses the mark bit vector generated by the mark phase to identify the chunks of heap storage that can be reclaimed for future allocations; these chunks are added to the free list.

Figure 1. Heap layout before and after garbage collection

The details of how the different GC phases work is beyond the scope of this article; my focus is on ensuring you understand the runtime characteristics. I encourage you to read the Diagnostics Guide (see Resources) for more information.

Figure 2 illustrates how the execution time is distributed between the application thread (or mutator) and GC threads. The horizontal axis is elapsed time and the vertical axis contains the threads, where n represents the number of processors on the machine. For this illustration, assume that the application is using one thread per processor. GC is illustrated by the blue boxes, which show stopped mutators and GC threads running. These collections consume 100 percent of the CPU's resources, and the mutator threads are idle. The figure overgeneralizes a bit so that we can compare this policy to other policies in this article. In actuality, GC duration and frequency vary depending on application and workload.

Figure 2. Distribution of CPU time between mutators and GC threads in the optthruput policy

Mutator vs. GC threads

A mutator thread is the application program that allocates objects. Another word for mutator isapplication. A GC thread is part of the memory management and performs the garbage collection.

Heap lock and thread allocation caches

The optthruput policy uses a contiguous heap area that all threads in the application share. A thread needs exclusive access to the heap for reserving space for new objects. This lock, called theheap lock, ensures that only one thread at a time can allocate an object. On machines with multiple CPUs, this lock can cause scaling problems because multiple allocation requests can occur at the same time, but each needs to have exclusive access to the heap lock.

To reduce this problem, each thread reserves a small chunk of memory called a thread allocation cache (also referred to asthread local heap, or TLH). This piece of storage is exclusive to a thread, so while allocating from it, the heap lock is not used. When the allocation cache is full, the thread goes back to the heap and asks for a new one, using the heap lock.

Fragmentation of the heap can prevent the thread from getting large TLHs, so a TLH fills quickly, causing the application thread to come back frequently to the heap for a new one. In this situation, the heap lock becomes a bottleneck; in such situations, thegencon or subpool policies can provide good alternatives.

optavgpause

For many applications, throughput is not as important as fast response time. Consider an application that processes work items that cannot take longer than 100 milliseconds to complete. With GC pauses in the 100 millisecond range, you get items that cannot be processed within this timeframe. A problem with garbage collection is that the pause time increases the maximum time it takes to process an item. Large heap sizes (available on 64-bit platforms) increase this effect because more objects are processed.

optavgpause is an alternative GC policy designed to keep pauses to a minimum. It does not guarantee a particular pause time, but pauses are shorter than those produced by the default GC policy. The idea is to perform some garbage collection work concurrently while the application is running. This is done in two places:

Concurrent mark and sweep: Before the heap is filled up, each mutator helps out and mark sobjects (concurrent mark). There is still a stop-the-world GC, but the pause is significantly shorter. After GC, the mutator threads help out and sweep objects (concurrent sweep).
Background GC thread: One (or more) low-priority background GC threads perform marking while the application is idle.

There is a throughput performance penalty of 5 to 10 percent over the default GC policy, depending on the application.

Figure 3 illustrates how the execution time can be divided between GC threads and mutator threads using optavgpause. The background tracing thread is not shown because it should not affect application performance.

Figure 3. Distribution of CPU time between mutators and GC threads in the optavgpause policy

The gray areas in the figure imply that concurrent tracing is enabled and that each mutator thread has to give up some of its processing time. Each concurrent phase is followed by a full garbage collection that completes the marking and sweeping that did not happen during the concurrent phase. The pause that results from this should be much shorter than the normal GC seen in optthruput, as indicated by a smaller box on the timescale in Figure 3. The gap between the end of the GC and the start of the concurrent phase vary, but during this phase, there is no significant performance impact.

gencon

A generational garbage collection strategy considers the lifetime of objects and places them in separate areas of the heap. In this way, it tries to overcome the drawbacks of a single heap in applications where most objects die young -- that is, where they do not survive many garbage collections.

With generational GC, objects that tend to survive for a long time are treated differently from short-lived objects. The heap is split into a nursery and a tenured area, as illustrated in Figure 4. Objects are created in the nursery and, if they live long enough, are promoted to the tenured area. Objects are promoted after having survived a certain number of garbage collections. The idea is that most objects are short-lived; by collecting the nursery frequently, these objects can be freed up without paying the cost of collecting the entire heap. The tenured area is garbage collected less often.

Figure 4. New and old area in gencon GC

As you can see in Figure 4, the nursery is in turn split into two spaces: allocate and survivor. Objects are allocated into the allocate space and, when that fills up, live objects are copied into the survivor space or into the tenured space, depending on their age. The spaces in the nursery then switch use, with allocate becoming survivor and survivor becoming allocate. The space occupied by dead objects can simply be overwritten by new allocations. Nursery collection is called a scavenge; Figure 5 illustrates what happens during this process:

Figure 5. Example of heap layout before and after GC

When the allocate space is full, garbage collection is triggered. Live objects are then traced and copied into the survivor space. This process is really inexpensive if most of the objects are dead. Furthermore, objects that have reached a copy threshold count are promoted into the tenured space. The object is then said to be tenured.

As the name Generational concurrent implies, the gencon policy has a concurrent aspect to it. The tenured space is concurrently marked with an approach similar to the one used in the optavgpause policy, except without concurrent sweep. All allocations pay a small throughput tax during the concurrent phase. With this approach, the pause time incurred from the tenure space collections is kept small.

Figure 6 shows how the execution time maps out when running gencon GC:

Figure 6. Distribution of CPU time between mutators and GC threads in gencon

A scavenge is short (shown by the small red boxes). Gray indicates that concurrent tracing starts followed by a collection of the tenured space, some of which happens concurrently. This is called a global collection, and it includes both a scavenge and a tenure space collection. How often a global collection occurs depends on the heap sizes and object lifetimes. The tenured space collection should be relatively quick because most of it has been collected concurrently.

subpool

The subpool policy can help increase performance on multiprocessor systems. As I mentioned earlier, this policy is available only on IBM pSeries and zSeries machines. The heap layout is the same as that for the optthruput policy, but the structure of the free list is different. Rather than having one free list for the entire heap, there are multiple lists, known as subpools. Each pool has an associated size by which the pools are ordered. An allocation request of a certain size can quickly be satisfied by going to the pool with that size. Atomic (platform-dependent) high-performing instructions are used to pop a free list entry off the list, avoiding serialized access. Figure 7 shows how the free chunks of storage are organized by size:

Figure 7. Subpool free chunks ordered by size

When the JVMs start or when a compaction has occurred, the subpools are not used because there are large areas of the heap free. In these situations, each processor gets its own dedicated mini-heap to satisfy requests. When the first garbage collection occurs, the sweep phase starts populating the subpools, and subsequent allocations mainly use subpools.

The subpool policy can reduce the time it takes to allocate objects. Atomic instructions ensure that allocations happen without acquiring a global heap lock. Mini-heaps local to a processor increase efficiency because cache interference is reduced. This has a direct effect on scalability, especially on multiprocessor systems. On platforms where subpool is not available, generational GC can provide similar benefits.

Conclusion

This article has highlighted the different GC policy choices available in the IBM SDK 5.0 and some of their characteristics. The default policy is sufficient for most applications; however, in some situations, other policies perform better. I have described some general scenarios in which you should consider switching to optavgpause, gencon, or subpool. Measuring application performance when evaluating a policy is very important, and Part 2 demonstrates this evaluation process in more detail.

clone() and the Cloneable Interface in Java

2012-10-16T00:32:00.001-07:00

Most of the methods defined by Object are discussed elsewhere in this book. However, one deserves special attention: clone( ). The clone( ) method generates a duplicate copy of the object on which it is called. Only classes that implement the Cloneable interface can be cloned.

The Cloneable interface defines no members. It is used to indicate that a class allows a bitwise copy of an object (that is, a clone) to be made. If you try to call clone( ) on a class that does not implementCloneable, a CloneNotSupportedException is thrown. When a clone is made, the constructor for the object being cloned is not called. A clone is simply an exact copy of the original.

Cloning is a potentially dangerous action, because it can cause unintended side effects. For example, if the object being cloned contains a reference variable called obRef, then when the clone is made, obRefin the clone will refer to the same object as does obRef in the original. If the clone makes a change to the contents of the object referred to by obRef, then it will be changed for the original object, too. Here is another example. If an object opens an I/O stream and is then cloned, two objects will be capable of operating on the same stream. Further, if one of these objects closes the stream, the other object might still attempt to write to it, causing an error.

Because cloning can cause problems, clone( ) is declared as protected inside Object. This means that it must either be called from within a method defined by the class that implements Cloneable, or it must be explicitly overridden by that class so that it is public. Let's look at an example of each approach.

The following program implements Cloneable and defines the method cloneTest( ), which calls clone( ) in Object:

// Demonstrate the clone() method.
class TestClone implements Cloneable {
int a;
double b;
// This method calls Object's clone().
TestClone cloneTest() {
try {
// call clone in Object.
return (TestClone) super.clone();
} catch(CloneNotSupportedException e) {
System.out.println("Cloning not allowed.");
return this;
}
}
}

class CloneDemo {
public static void main(String args[]) {
TestClone x1 = new TestClone();
TestClone x2;
x1.a = 10;
x1.b = 20.98;
x2 = x1.cloneTest(); // clone x1
System.out.println("x1: " + x1.a + " " + x1.b);
System.out.println("x2: " + x2.a + " " + x2.b);
}
}

Here, the method cloneTest( ) calls clone( ) in Object and returns the result. Notice that the object returned by clone( ) must be cast into its appropriate type (TestClone). The following example overrides clone( ) so that it can be called from code outside of its class. To do this, its access specifier must be public, as shown here:

// Override the clone() method.
class TestClone implements Cloneable {
int a;
double b;
// clone() is now overridden and is public.
public Object clone() {
try {
// call clone in Object.
return super.clone();
} catch(CloneNotSupportedException e) {
System.out.println("Cloning not allowed.");
return this;
}
}
}

class CloneDemo2 {
public static void main(String args[]) {
TestClone x1 = new TestClone();
TestClone x2;
x1.a = 10;
x1.b = 20.98;
// here, clone() is called directly.
x2 = (TestClone) x1.clone();
System.out.println("x1: " + x1.a + " " + x1.b);
System.out.println("x2: " + x2.a + " " + x2.b);
}
}

The side effects caused by cloning are sometimes difficult to see at first. It is easy to think that a class is safe for cloning when it actually is not. In general, you should not implement Cloneable for any class without good reason.

Analytic functions by Example

2012-10-08T22:01:00.001-07:00

This article provides a clear, thorough concept of analytic functions and its various options by a series of simple yet concept building examples. The article is intended for SQL coders, who for might be not be using analytic functions due to unfamiliarity with its cryptic syntax or uncertainty about its logic of operation. Often I see that people tend to reinvent the feature provided by analytic functions by native join and sub-query SQL. This article assumes familiarity with basic Oracle SQL, sub-query, join and group function from the reader. Based on that familiarity, it builds the concept of analytic functions through a series of examples.

It is true that whatever an analytic function does can be done by native SQL, with join and sub-queries. But the same routine done by analytic function is always faster, or at least as fast, when compared to native SQL. Moreover, I am not considering here the amount of time that is spent in coding the native SQLs, testing, debugging and tuning them.

The general syntax of analytic function is:

Function(arg1,..., argn) OVER ( [PARTITION BY <...>] [ORDER BY <....>] [<window_clause>] )

<window_clause> is like "ROW <?>" or "RANK <?>"
All the keywords will be dealt in details as we walk through the examples. The script for creating the schema (SCOTT) on which the example queries of this article are run can be obtained in ORACLE_HOME/sqlplus/demo/demobld.sql of any standard Oracle installation.

How are analytic functions different from group or aggregate functions?

SELECT deptno,
COUNT(*) DEPT_COUNT
FROM emp
WHERE deptno IN (20, 30)
GROUP BY deptno;

DEPTNO                 DEPT_COUNT             
---------------------- ---------------------- 
20                     5                      
30                     6                      

2 rows selected

Query-1

Consider the Query-1 and its result. Query-1 returns departments and their employee count. Most importantly it groups the records into departments in accordance with the GROUP BY clause. As such any non-"group by" column is not allowed in the select clause.

SELECT empno, deptno, 
COUNT(*) OVER (PARTITION BY 
deptno) DEPT_COUNT
FROM emp
WHERE deptno IN (20, 30);

     EMPNO     DEPTNO DEPT_COUNT
---------- ---------- ----------
      7369         20          5
      7566         20          5
      7788         20          5
      7902         20          5
      7876         20          5
      7499         30          6
      7900         30          6
      7844         30          6
      7698         30          6
      7654         30          6
      7521         30          6

11 rows selected.

Query-2

Now consider the analytic function query (Query-2) and its result. Note the repeating values of DEPT_COUNT column.

This brings out the main difference between aggregate and analytic functions. Though analytic functions give aggregate result they do not group the result set. They return the group value multiple times with each record. As such any other non-"group by" column or expression can be present in the select clause, for example, the column EMPNO inQuery-2.

Analytic functions are computed after all joins, WHERE clause, GROUP BY and HAVING are computed on the query. The main ORDER BY clause of the query operates after the analytic functions. So analytic functions can only appear in the select list and in the main ORDER BY clause of the query.

In absence of any PARTITION or <window_clause> inside the OVER( ) portion, the function acts on entire record set returned by the where clause. Note the results of Query-3 and compare it with the result of aggregate function query Query-4.

SELECT empno, deptno, 
COUNT(*) OVER ( ) CNT
FROM emp
WHERE deptno IN (10, 20)
ORDER BY 2, 1;

     EMPNO     DEPTNO        CNT
---------- ---------- ----------
      7782         10          8
      7839         10          8
      7934         10          8
      7369         20          8
      7566         20          8
      7788         20          8
      7876         20          8
      7902         20          8

Query-3

SELECT COUNT(*) FROM emp
WHERE deptno IN (10, 20);

  COUNT(*)
----------
         8

Query-4

How to break the result set in groups or partitions?

It might be obvious from the previous example that the clause PARTITION BY is used to break the result set into groups. PARTITION BY can take any non-analytic SQL expression.

Some functions support the <window_clause> inside the partition to further limit the records they act on. In the absence of any <window_clause> analytic functions are computed on all the records of the partition clause.

The functions SUM, COUNT, AVG, MIN, MAX are the common analytic functions the result of which does not depend on the order of the records.

Functions like LEAD, LAG, RANK, DENSE_RANK, ROW_NUMBER, FIRST, FIRST VALUE, LAST, LAST VALUE depends on order of records. In the next example we will see how to specify that.

How to specify the order of the records in the partition?

The answer is simple, by the "ORDER BY" clause inside the OVER( ) clause. This is different from the ORDER BY clause of the main query which comes after WHERE. In this section we go ahead and introduce each of the very useful functions LEAD, LAG, RANK, DENSE_RANK, ROW_NUMBER, FIRST, FIRST VALUE, LAST, LAST VALUE and show how each depend on the order of the record.

The general syntax of specifying the ORDER BY clause in analytic function is:

ORDER BY <sql_expr> [ASC or DESC] NULLS [FIRST or LAST]

The syntax is self-explanatory.

ROW_NUMBER, RANK and DENSE_RANK

All the above three functions assign integer values to the rows depending on their order. That is the reason of clubbing them together.

ROW_NUMBER( ) gives a running serial number to a partition of records. It is very useful in reporting, especially in places where different partitions have their own serial numbers. In Query-5, the function ROW_NUMBER( ) is used to give separate sets of running serial to employees of departments 10 and 20 based on their HIREDATE.

SELECT empno, deptno, hiredate,
ROW_NUMBER( ) OVER (PARTITION BY
deptno ORDER BY hiredate
NULLS LAST) SRLNO
FROM emp
WHERE deptno IN (10, 20)
ORDER BY deptno, SRLNO;

EMPNO  DEPTNO HIREDATE       SRLNO
------ ------- --------- ----------
  7782      10 09-JUN-81          1
  7839      10 17-NOV-81          2
  7934      10 23-JAN-82          3
  7369      20 17-DEC-80          1
  7566      20 02-APR-81          2
  7902      20 03-DEC-81          3
  7788      20 09-DEC-82          4
  7876      20 12-JAN-83          5

8 rows selected.

Query-5 (ROW_NUMBER example)

RANK and DENSE_RANK both provide rank to the records based on some column value or expression. In case of a tie of 2 records at position N, RANK declares 2 positions N and skips position N+1 and gives position N+2 to the next record. While DENSE_RANK declares 2 positions N but does not skip position N+1.

Query-6 shows the usage of both RANK and DENSE_RANK. For DEPTNO 20 there are two contenders for the first position (EMPNO 7788 and 7902). Both RANK and DENSE_RANK declares them as joint toppers. RANK skips the next value that is 2 and next employee EMPNO 7566 is given the position 3. For DENSE_RANK there are no such gaps.

SELECT empno, deptno, sal,
RANK() OVER (PARTITION BY deptno
ORDER BY sal DESC NULLS LAST) RANK,
DENSE_RANK() OVER (PARTITION BY
deptno ORDER BY sal DESC NULLS
LAST) DENSE_RANK
FROM emp
WHERE deptno IN (10, 20)
ORDER BY 2, RANK;

EMPNO  DEPTNO   SAL  RANK DENSE_RANK
------ ------- ----- ----- ----------
  7839      10  5000     1          1
  7782      10  2450     2          2
  7934      10  1300     3          3
  7788      20  3000     1          1
  7902      20  3000     1          1
  7566      20  2975     3          2
  7876      20  1100     4          3
  7369      20   800     5          4

8 rows selected.

Query-6 (RANK and DENSE_RANK example)

LEAD and LAG

LEAD has the ability to compute an expression on the next rows (rows which are going to come after the current row) and return the value to the current row. The general syntax of LEAD is shown below:

LEAD (<sql_expr>, <offset>, <default>) OVER (<analytic_clause>)

<sql_expr> is the expression to compute from the leading row.
<offset> is the index of the leading row relative to the current row.
<offset> is a positive integer with default 1.
<default> is the value to return if the <offset> points to a row outside the partition range.

The syntax of LAG is similar except that the offset for LAG goes into the previous rows.
Query-7 and its result show simple usage of LAG and LEAD function.

SELECT deptno, empno, sal,
LEAD(sal, 1, 0) OVER (PARTITION BY dept ORDER BY sal DESC NULLS LAST) NEXT_LOWER_SAL,
LAG(sal, 1, 0) OVER (PARTITION BY dept ORDER BY sal DESC NULLS LAST) PREV_HIGHER_SAL
FROM emp
WHERE deptno IN (10, 20)
ORDER BY deptno, sal DESC;

 DEPTNO  EMPNO   SAL NEXT_LOWER_SAL PREV_HIGHER_SAL
------- ------ ----- -------------- ---------------
     10   7839  5000           2450               0
     10   7782  2450           1300            5000
     10   7934  1300              0            2450
     20   7788  3000           3000               0
     20   7902  3000           2975            3000
     20   7566  2975           1100            3000
     20   7876  1100            800            2975
     20   7369   800              0            1100

8 rows selected.

Query-7 (LEAD and LAG)

FIRST VALUE and LAST VALUE function

The general syntax is:

FIRST_VALUE(<sql_expr>) OVER (<analytic_clause>)

The FIRST_VALUE analytic function picks the first record from the partition after doing the ORDER BY. The <sql_expr> is computed on the columns of this first record and results are returned. The LAST_VALUE function is used in similar context except that it acts on the last record of the partition.

-- How many days after the first hire of each department were the next
-- employees hired?

SELECT empno, deptno, hiredate ? FIRST_VALUE(hiredate)
OVER (PARTITION BY deptno ORDER BY hiredate) DAY_GAP
FROM emp
WHERE deptno IN (20, 30)
ORDER BY deptno, DAY_GAP;

     EMPNO     DEPTNO    DAY_GAP
---------- ---------- ----------
      7369         20          0
      7566         20        106
      7902         20        351
      7788         20        722
      7876         20        756
      7499         30          0
      7521         30          2
      7698         30         70
      7844         30        200
      7654         30        220
      7900         30        286

11 rows selected.

Query-8 (FIRST_VALUE)

FIRST and LAST function

The FIRST function (or more properly KEEP FIRST function) is used in a very special situation. Suppose we rank a group of record and found several records in the first rank. Now we want to apply an aggregate function on the records of the first rank. KEEP FIRST enables that.

The general syntax is:

Function( ) KEEP (DENSE_RANK FIRST ORDER BY <expr>) OVER (<partitioning_clause>)

Please note that FIRST and LAST are the only functions that deviate from the general syntax of analytic functions. They do not have the ORDER BY inside the OVER clause. Neither do they support any <window> clause. The ranking done in FIRST and LAST is always DENSE_RANK. The query below shows the usage of FIRST function. The LAST function is used in similar context to perform computations on last ranked records.

-- How each employee's salary compare with the average salary of the first
-- year hires of their department?

SELECT empno, deptno, TO_CHAR(hiredate,'YYYY') HIRE_YR, sal,
TRUNC(
AVG(sal) KEEP (DENSE_RANK FIRST
ORDER BY TO_CHAR(hiredate,'YYYY') )
OVER (PARTITION BY deptno)
     ) AVG_SAL_YR1_HIRE
FROM emp
WHERE deptno IN (20, 10)
ORDER BY deptno, empno, HIRE_YR;

     EMPNO     DEPTNO HIRE        SAL AVG_SAL_YR1_HIRE
---------- ---------- ---- ---------- ----------------
      7782         10 1981       2450             3725
      7839         10 1981       5000             3725
      7934         10 1982       1300             3725
      7369         20 1980        800              800
      7566         20 1981       2975              800
      7788         20 1982       3000              800
      7876         20 1983       1100              800
      7902         20 1981       3000              800

8 rows selected.

Query-9 (KEEP FIRST)

How to specify the Window clause (ROW type or RANGE type windows)?

Some analytic functions (AVG, COUNT, FIRST_VALUE, LAST_VALUE, MAX, MIN and SUM among the ones we discussed) can take a window clause to further sub-partition the result and apply the analytic function. An important feature of the windowing clause is that it is dynamic in nature.

The general syntax of the <window_clause> is

[ROW or RANGE] BETWEEN <start_expr> AND <end_expr>

<start_expr> can be any one of the following

UNBOUNDED PECEDING
CURRENT ROW
<sql_expr> PRECEDING or FOLLOWING.

UNBOUNDED FOLLOWING or
CURRENT ROW or
<sql_expr> PRECEDING or FOLLOWING.

For ROW type windows the definition is in terms of row numbers before or after the current row. So for ROW type windows <sql_expr> must evaluate to a positive integer.

For RANGE type windows the definition is in terms of values before or after the current ORDER. We will take this up in details latter.

The ROW or RANGE window cannot appear together in one OVER clause. The window clause is defined in terms of the current row. But may or may not include the current row. The start point of the window and the end point of the window can finish before the current row or after the current row. Only start point cannot come after the end point of the window. In case any point of the window is undefined the default is UNBOUNDED PRECEDING for <start_expr> and UNBOUNDED FOLLOWING for <end_expr>.

If the end point is the current row, syntax only in terms of the start point can be can be
[ROW or RANGE] [<start_expr> PRECEDING or UNBOUNDED PRECEDING ]

[ROW or RANGE] CURRENT ROW is also allowed but this is redundant. In this case the function behaves as a single-row function and acts only on the current row.

ROW Type Windows

For analytic functions with ROW type windows, the general syntax is:

Function( ) OVER (PARTITIN BY <expr1> ORDER BY <expr2,..> ROWS BETWEEN <start_expr> AND <end_expr>)
or
Function( ) OVER (PARTITON BY <expr1> ORDER BY <expr2,..> ROWS [<start_expr> PRECEDING or UNBOUNDED PRECEDING]

For ROW type windows the windowing clause is in terms of record numbers.

The query Query-10 has no apparent real life description (except column FROM_PU_C) but the various windowing clause are illustrated by a COUNT(*) function. The count simply shows the number of rows inside the window definition. Note the build up of the count for each column for the YEAR 1981.

The column FROM_P3_TO_F1 shows an example where start point of the window is before the current row and end point of the window is after current row. This is a 5 row window; it shows values less than 5 during the beginning and end.

-- The query below has no apparent real life description (except 
-- column FROM_PU_C) but is remarkable in illustrating the various windowing
-- clause by a COUNT(*) function.
 
SELECT empno, deptno, TO_CHAR(hiredate, 'YYYY') YEAR,
COUNT(*) OVER (PARTITION BY TO_CHAR(hiredate, 'YYYY')
ORDER BY hiredate ROWS BETWEEN 3 PRECEDING AND 1 FOLLOWING) FROM_P3_TO_F1,
COUNT(*) OVER (PARTITION BY TO_CHAR(hiredate, 'YYYY')
ORDER BY hiredate ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) FROM_PU_TO_C,
COUNT(*) OVER (PARTITION BY TO_CHAR(hiredate, 'YYYY')
ORDER BY hiredate ROWS BETWEEN 3 PRECEDING AND 1 PRECEDING) FROM_P2_TO_P1,
COUNT(*) OVER (PARTITION BY TO_CHAR(hiredate, 'YYYY')
ORDER BY hiredate ROWS BETWEEN 1 FOLLOWING AND 3 FOLLOWING) FROM_F1_TO_F3
FROM emp
ORDEDR BY hiredate

 EMPNO  DEPTNO YEAR FROM_P3_TO_F1 FROM_PU_TO_C FROM_P2_TO_P1 FROM_F1_TO_F3
------ ------- ---- ------------- ------------ ------------- -------------
  7369      20 1980             1            1             0             0
  <font bgcolor=yellow>7499      30 1981             2            1             0             3
  7521      30 1981             3            2             1             3
  7566      20 1981             4            3             2             3
  7698      30 1981             5            4             3             3
  7782      10 1981             5            5             3             3
  7844      30 1981             5            6             3             3
  7654      30 1981             5            7             3             3
  7839      10 1981             5            8             3             2
  7900      30 1981             5            9             3             1
  7902      20 1981             4           10             3             0</font>
  7934      10 1982             2            1             0             1
  7788      20 1982             2            2             1             0
  7876      20 1983             1            1             0             0

14 rows selected.

Query-10 (ROW type windowing example)

The column FROM_PU_TO_CURR shows an example where start point of the window is before the current row and end point of the window is the current row. This column only has some real world significance. It can be thought of as the yearly employee build-up of the organization as each employee is getting hired.

The column FROM_P2_TO_P1 shows an example where start point of the window is before the current row and end point of the window is before the current row. This is a 3 row window and the count remains constant after it has got 3 previous rows.

The column FROM_F1_TO_F3 shows an example where start point of the window is after the current row and end point of the window is after the current row. This is a reverse of the previous column. Note how the count declines during the end.

RANGE Windows

For RANGE windows the general syntax is same as that of ROW:

Function( ) OVER (PARTITION BY <expr1> ORDER BY <expr2> RANGE BETWEEN <start_expr> AND <end_expr>)
or
Function( ) OVER (PARTITION BY <expr1> ORDER BY <expr2> RANGE [<start_expr> PRECEDING or UNBOUNDED PRECEDING]

For <start_expr> or <end_expr> we can use UNBOUNDED PECEDING, CURRENT ROW or <sql_expr> PRECEDING or FOLLOWING. However for RANGE type windows <sql_expr> must evaluate to value compatible with ORDER BY expression <expr1>.

<sql_expr> is a logical offset. It must be a constant or expression that evaluates to a positive numeric value or an interval literal. Only one ORDER BY expression is allowed.

If <sql_expr> evaluates to a numeric value, then the ORDER BY expr must be a NUMBER or DATE datatype. If <sql_expr> evaluates to an interval value, then the ORDER BY expr must be a DATE datatype.

Note the example (Query-11) below which uses RANGE windowing. The important thing here is that the size of the window in terms of the number of records can vary.

-- For each employee give the count of employees getting half more that their 
-- salary and also the count of employees in the departments 20 and 30 getting half 
-- less than their salary.
 
SELECT deptno, empno, sal,
Count(*) OVER (PARTITION BY deptno ORDER BY sal RANGE
BETWEEN UNBOUNDED PRECEDING AND (sal/2) PRECEDING) CNT_LT_HALF,
COUNT(*) OVER (PARTITION BY deptno ORDER BY sal RANGE
BETWEEN (sal/2) FOLLOWING AND UNBOUNDED FOLLOWING) CNT_MT_HALF
FROM emp
WHERE deptno IN (20, 30)
ORDER BY deptno, sal

 DEPTNO  EMPNO   SAL CNT_LT_HALF CNT_MT_HALF
------- ------ ----- ----------- -----------
     20   7369   800           0           3
     20   7876  1100           0           3
     20   7566  2975           2           0
     20   7788  3000           2           0
     20   7902  3000           2           0
     30   7900   950           0           3
     30   7521  1250           0           1
     30   7654  1250           0           1
     30   7844  1500           0           1
     30   7499  1600           0           1
     30   7698  2850           3           0

11 rows selected.

Query-11 (RANGE type windowing example)

Order of computation and performance tips

Defining the PARTITOIN BY and ORDER BY clauses on indexed columns (ordered in accordance with the PARTITION CLAUSE and then the ORDER BY clause in analytic function) will provide optimum performance. For Query-5, for example, a composite index on (deptno, hiredate) columns will prove effective.

It is advisable to always use CBO for queries using analytic functions. The tables and indexes should be analyzed and optimizer mode should be CHOOSE.

Even in absence of indexes analytic functions provide acceptable performance but need to do sorting for computing partition and order by clause. If the query contains multiple analytic functions, sorting and partitioning on two different columns should be avoided if they are both not indexed.

Conclusion

The aim of this article is not to make the reader try analytic functions forcibly in every other complex SQL. It is meant for a SQL coder, who has been avoiding analytic functions till now, even in complex analytic queries and reinventing the same feature much painstakingly by native SQL and join query. Its job is done if such a person finds analytic functions clear, understandable and usable after going through the article, and starts using them.

Jmeter Sampler Client

2012-04-30T01:39:00.001-07:00

Download latest Jmeter binaries and install as for instruction given.

You can use some Java development tools like eclipse for developing client program.

i am just creating simple sample program using eclipse because it provide lot of utility like auto compile, create jar etc.

Lets start with creating simple java class JmeterTestClient.java inside jmeter.sampler either by implementing JavaSamplerClient(org.apache.jmeter.protocol.java.sampler.JavaSamplerClient) or extending AbstractJavaSamplerClient(org.apache.jmeter.protocol.java.sampler.AbstractJavaSamplerClient)

you need ApacheJMeter_java.jar ApacheJMeter_core.jar for compilation only. You can find both jars at <<Installation_dir>>\jakarta-jmeter-2.4\lib\ext

When possible, Java tests should extend AbstractJavaSamplerClient rather than implementing JavaSamplerClient directly. This should protect your tests from future changes to the interface. While it may be necessary to make changes to the JavaSamplerClient interface from time to time (therefore requiring changes to any implementations of this interface), we intend to make this abstract class provide reasonable default implementations of any new methods so that subclasses do not necessarily need to be updated for new versions. Implementing JavaSamplerClient directly will continue to be supported for cases where extending this class is not possible (for example, when the client class is already a subclass of some other class).

This interface defines the interactions between the JavaSampler and external Java programs which can be executed by JMeter. Any Java class which wants to be executed as a JMeter test must implement this interface (either directly or indirectly through AbstractJavaSamplerClient).

JMeter will create one instance of a JavaSamplerClient implementation for each user/thread in the test. Additional instances may be created for internal use by JMeter (for example, to find out what parameters are supported by the client).

When the test is started, setupTest() will be called on each thread's JavaSamplerClient instance to initialize the client. Then runTest() will be called for each iteration of the test. Finally, teardownTest() will be called to allow the client to do any necessary clean-up

Compile the above java file and export as jar and put inside <<Installation_dir>>\jakarta-jmeter-2.4\lib\ext

start jmeter , select testplan creat new thread group and select thread group add java request sampler . select JmeterTestClient from classname drop down . save your test plan and run.

in my example i printed some text are you can see in console ,You can set loop count in thread group.
The sample code is.

package jmeter.sampler;

import org.apache.jmeter.config.Arguments;
import org.apache.jmeter.protocol.java.sampler.JavaSamplerClient;
import org.apache.jmeter.protocol.java.sampler.JavaSamplerContext;
import org.apache.jmeter.samplers.SampleResult;

public class JmeterTestClient implements JavaSamplerClient {
int count;

@Override
public Arguments getDefaultParameters() {
count = 10;
System.out.println("inside default");
return null;
}

@Override
public SampleResult runTest(JavaSamplerContext arg0) {
System.out.println("Inside Run test:" + count++);
return null;
}

@Override
public void setupTest(JavaSamplerContext arg0) {
count++;
System.out.println("inside Setup");
}

@Override
public void teardownTest(JavaSamplerContext arg0) {
System.out.println("inside tear Down:" + count);
}

}

Understanding how EJB calls operate in WebSphere Application Server V6.1

2012-04-26T21:55:00.000-07:00

Summary: Many applications use Enterprise JavaBeans™ (EJB) deployed in an IBM® WebSphere ® Application Server EJB container. Communication to these EJB components requires the use of concepts like Internet Inter-ORB Protocol (IIOP) communication, JNDI look-ups, and workload management. This interaction can be complex and is often a black box for many WebSphere users. This article provides a basic understanding of how EJB communication works in the context of WebSphere Application Server. Communication into and out of the ORB is addressed, along with the definitions of basic call patterns and components involved in making calls to EJB components deployed in the EJB container. A basic understanding of J2EE™ principles and general administrative knowledge of WebSphere Application Server is assumed.

Introduction

For many WebSphere users, EJB communications are a black box. That is, we understand that calling method 'X' will return the desired result from the EJB business logic. Under normal operating conditions, this knowledge may be sufficient as it is not usually necessary to understand the inner workings of the ORB. However, when performance problems or load balancing issues arise, in-depth knowledge of the ORB can greatly enhance the the problem determination effort and reduce time to resolution. As with any WebSphere component, understanding the mechanism of operation for the ORB and its related communication protocols can yield further expertise when troubleshooting it.

The ORB

The ORB is the central part of CORBA, or Common Object Request Broker Architecture, which aims to make remote objects usable like local objects in object-oriented, distributed applications. In addition, ORB lets these objects communicate with each other regardless of the specific platforms, techniques, and languages that are used to implement those objects.

CORBA is an open, vendor-neutral specification for distributed computing created by the Object Management Group (OMG), a consortium of over 600 software vendors and other companies promoting open, standard interfaces. CORBA and its Interface Definition Language (IDL) provides a common format for representing an object that can be distributed to applications written in Java™, C, C++, and other languages and running on different operating systems and machines.

The ORB (Object Request Broker) plays an integral part in the application server runtime. The ORB is implemented as part of the Java Virtual Machine (JVM). IBM WebSphere Application Server includes additional code (generally referred to as the extension ORB) that manages the ORB instance, and also provides the plug-in framework for object services (such as security). This IBM ORB implementation is not the same as the Sun™ ORB; the code base is different altogether. It is the foundation on which the EJB container depends to broker communication for Java Remote Method Invocation (RMI) interactions. RMI is Java’s traditional method of performing remote communications and uses a non-standardized protocol, named Java Remote Method Protocol (JRMP), to communicate between Java objects. At the most basic level, RMI is just Java’s object-oriented version of Remote Procedure Calls (RPCs). WebSphere Application Server uses the ORB to facilitate client/server RMI communication, as well as to communicate between components.

The ORB manages inbound and outbound requests for remote Java objects, like EJB components. ORB provides a framework for clients to locate EJBs on the server and invoke operations on those as if they were local. (This concept is referred to as local-remote transparency.) The relationship of the ORB with the EJB container can be described with Figure 1.

Figure 1. ORB and the EJB container

Inter-ORB communication

Communication between ORBs is accomplished via the IIOP, which is a concrete implementation of the General Inter-Orb Protocol (GIOP) abstract reference specification. The Interoperable Object Reference (IOR) is simply a string representation of the CORBA or RMI-IIOP object reference in a format that ORBs can understand. You can think of this as a normal Java object reference used in everyday coding. IIOP maps GIOP to the TCP/IP communication stack.

The GIOP protocol specifies several different types of messages, such as request messages, reply messages, locate request messages, locate reply messages, and fragment messages. The request and reply message types are self-explanatory, as they deal with the request and response interaction between client and server applications. The locate request and fragment message types might not be as familiar to you unless you have needed to read through GIOP tracing to solve a problem:

The locate request message type is used to verify that the ORB understands and houses the requested remote object. If the ORB does not know the object requested in the locate request message, it will attempt to provide details (for example, an OBJECT_FORWARD which points to a server that does house the object) on where the client can find the requested object.
The fragment message is a continuation of the previous message delivered to the ORB.

Processing large GIOP messages can cause problems for the ORB. GIOP messages can be fragmented into manageable pieces for easier and faster processing. Custom properties can be set in WebSphere Application Server that specify the exact size in which to fragment a message (or whether or not to fragment messages at all). By default, WebSphere Application Server will fragment GIOP messages at 1024 bytes.

In the most basic of interaction scenarios, the client and the server ORBs communicate directly with each other to satisfy application requests for remote resources. However, in more complex interactions, such as clustered servers with workload management features, processes other than the client and server can become involved (for the purpose of this article) in the NodeAgent process.

Each application server contains an ORB that listens for incoming requests. The ORB has three or more ports that are associated with it: the ORB listener port, the bootstrap port, and the security ports. The ORB listener port is the actual port that the ORB listens on for incoming IIOP requests if ORB security is not enabled. The bootstrap port is always an active, well-advertised port, and always returns an indirect IOR containing the secured and unsecured listener ports. A well-constructed client should always bootstrap to the server in which the remote resource resides (or to a multiple host provider URL that points to the cluster if the resource resides in a cluster). There are three other ports associated with the ORB and security: the CSIv2 Client Authentication SSL port, the CSIv2 SSL port, and the SAS SSL port. (Details of these ports is beyond the scope of this article and will not be covered here.) ORB listener ports are, by default, dynamically determined from the platform’s ephemeral port range at run time. However, these ports can be set to a specific static port for various reasons, such as firewall rules, business policies, and so on.

When an ORB client requests a reference to a remote object, the remote ORB will reply with an Interoperable Object Reference (IOR), which is simply a string representation of the CORBA or RMI-IIOP object reference in a format that ORBs can understand, and at the very basic level contains host and port information for the server on which the object can be found. The IOR can be thought of as a typical Java object reference used in everyday coding.

How is the NodeAgent involved?

First, the NodeAgent process is only involved in an ORB transaction in the WebSphere Application Server Network Deployment product; it is not present in the base application server offering. The NodeAgent process houses a service called the location service daemon, which enables application servers to register their services and remote objects by providing the daemon with direct IORs pointing to the said application server. Later, when a locate request comes into the location service daemon, it uses the registration information to provide a direct IOR back to a client that points to the registered application server’s ORB listener ports. This registration/IOR information could be thought of as a sort of routing table that is used by the daemon to locate EJBs and other resources found on the various application servers managed by a particular NodeAgent.

Before a managed application server can start properly, a NodeAgent must be available locally on the node. It is possible to configure the application servers such that the dependency on the NodeAgent process (actually the location service daemon service) is gone. However, this setup involves carrying out static routing for IIOP requests, and thus WebSphere Application Server’s workload management services would be nullified. This static routing setup should be avoided unless truly required. (Consult IBM Support before making this assessment on your own.)

In the event that the NodeAgent process dies or must be taken down or restarted during the operation of an application server on the same node with EJBs deployed on it, exceptions might be thrown in the server logs once the NodeAgent is unavailable. These messages are due to the fact that, once the NodeAgent is removed from the process, the IIOP communication is altered in a way that prevents both ORBs (client and server) from communicating correctly. However, as long as clients already have direct IORs to their desired servers, have already created their EJBHome object, and are making remote calls to that EJB, the NodeAgent is unlikely to be involved in the IIOP communication after that point.

Ideally, incoming locate requests to the location service daemon for EJB-related objects will not come in until all the application servers on a particular node have completed their registration with the location service daemon on the NodeAgent. When locate requests come in for objects on servers that have not yet registered or are temporarily down (and hence no longer registered), a CORBA.OBJECT_NOT_EXIST exception will be returned to the client. In this case, the status of the application server needs to be checked to ensure they are all up and running and have registered with the location service daemon.

Understanding the ORB call pattern

In order for high level application functionality (like EJB components) to work, inter-ORB communication must occur. There are two basic scenarios that we can observe:

A standalone application server (WebSphere Application Server, not Network Deployment)
A clustered application server in a WebSphere Application Server Network Deployment environment.

The next sections describe these scenarios.

WebSphere Application Server base instance

In this scenario, the EJB component and its client are contained in the separate application servers. The client code to make the call might look like this:

Listing 1

Hashtable env = new Hashtable();
env.put(Context.INITIAL_CONTEXT_FACTORY,"com.ibm.websphere.naming
.WsnInitialContextFactory");
env.put(Context.PROVIDER_URL,"corbaloc::isador:2810");
Context ctx = new InitialContext(env);
TestEJBHome home = (TestEJBHome)
PortableRemoteObject.narrow(ctx.lookup("ejb/ejbs/TestEJBHome"),
 TestEJBHome.class);
TestEJB bean = home.create();

The code in bold text are the lines that result in remote ORB calls. When put into motion, you can follow the events in Figure 2, where the events are listed, in sequence, from top to bottom.

Figure 2. ORB event calls in base instance

Figure 2 is divided into three parts, each part corresponding to a portion of the Java code in Listing 1:

The top section contains the events that result from the InitialContext creation on the client.
The middle section contains the events that result from the InitialContext.lookup() method.
The bottom section contains the events that result from the creation of the EJB instance.

From this diagram, you can see that the process of obtaining an initial context from the server involves sending a locate request message to the ORB (client and server ORB are the same here) on the ORB’s bootstrap port. The resulting reply message contains a direct IOR (with host and TCP/IP and SSL port information) that points the client to the server containing the requested object. Next, the client issues another locate request message, but this time to the ORB’s listener port. After this, the client requests some information from the ORB to establish an initial context.

After the InitialContext value has been obtained, the client code then does a lookup in the JNDI namespace to determine how to get a reference to the desired remote object (WLMTestEJB). To accomplish this, the client issues yet another locate request aimed at the ORB’s listener port. The ORB replies back to the client via a locate reply message stating that the object is indeed contained locally. In response to the locate reply, the client then requests that the ORB resolve the object and return a reference to the client. Under the covers, this entails the ORB using a tie class to connect the real Java object with the request that came to the ORB. When the client has the reference, it uses a “narrow” call to marshal the generic object returned from the ORB into the desired object type.

The last thing the code does is create an instance of the remote EJB. To complete the process, the client issues a locate request to the remote ORB, to which the ORB replies with a locate reply stating that the object is present. Finally, the client then invokes the create() method on the remote object. When the process is complete in the ORB, the ORB marshals the response back to the client. All data that goes into or leaves the ORB (like method parameters and method return values) are serialized and deserialized using a format called common data representation (CDR). As such, all values passed into the ORB must implement the serializable interface.

The locate request messages used in this base application server scenario are simple locate requests that are processed by the ORB itself. This is different from the next scenario because the location service daemon plug-in (attached to the ORB instance) handles messages responding to the locate requests.

WebSphere Application Server Network Deployment clustered instance

In this scenario, the major difference is that the client ORB and server ORB will almost always be two different JVM instances, often located on completely separate hosts. The process will appear much the same, but you will notice the involvement of the location service daemon server contained in the NodeAgent. Listing 2 shows the code involved to make the call.

Listing 2

Hashtable env = new Hashtable();
env.put(Context.INITIAL_CONTEXT_FACTORY,"com.ibm.websphere.naming
.WsnInitialContextFactory");
env.put(Context.PROVIDER_URL,"corbaloc::boris:9811,:natasha
:9812");
Context ctx = new InitialContext(env);
TestEJBHome home = (TestEJBHome)
PortableRemoteObject.narrow(ctx.lookup("ejb/ejbs/TestEJBHome"),
 TestEJBHome.class);
TestEJB bean = home.create();

The code only changed from the previous example in only one place: the values placed in the PROVIDER_URL key of the environment Hashtable object. In this case, the developer included multiple corbaloc URLs, each of which points to a separate server’s bootstrap port. This change introduces some fault tolerance and failover to the process as the code now has two paths from which the InitialContext object can be obtained. As in the previous scenario, the code in bold text is responsible for generating the remote ORB requests. Figure 3 illustrates the sequence of events for this scenario, formatted the same as the previous example.

Figure 3. ORB event calls in clustered instance

For the creation of the InitialContext object in the top section, there are no obvious differences between this scenario and the first one. However, with the addition of WebSphere Application Server workload management in the picture, there are slight differences.

First, during the creation of the InitialContext (top section), a list of IORs that point to each of the cell’s location service daemon clusters are prepared and returned to the client ORB during the getProperties() portion of the conversation. When the client ORB makes further calls to the server ORB, the workload management plug-in on the client side iterates over the list and issues a resolve_complete_info request. This first request is routed to the location service daemon service in the NodeAgent (shown as the first call in the middle section), where the desired cluster data is sent back to the client in the form of a direct IOR pointing to the destination server.

In the bottom section of the diagram, you can see that the locate request message issued in the first scenario is not needed and the create() method is invoked instead, immediately without having to process the additional locate request because workload management has already “located” an appropriate application server.

Summary

This article defined the ORB and how it provides the application server runtime with vital functionality needed to service clients making remote requests to an EJB server. The high level sequence of events that make up inter-ORB communications was also explained, along with the role that the NodeAgent plays in the part of ORB communication between clients and servers.

There is much more information available on these topics. If you wish to continue your reading, you might want to look into these subjects:

GIOP protocol
Problem determination for the ORB/EJB container
RMI over IIOP
Java Naming and Directory Interface (JNDI)
EJB clustering
WebSphere workload management (WLM)
CORBA and the OMG.

Acknowledgements

The authors extend their thanks to Paul Bullis and Claudia Barrett for providing guidance and clarification on the topics discussed in this article.

FTP Configuration on a Firewall

2012-04-18T02:52:00.002-07:00

This page gives a detailed description on what TCP activity goes on during a FTP session. If you are looking for answers to connection problems caused by routers and firewalls, you should look here.

Whenever an FTP client creates a connection to an FTP server at port 21, and starts to request data, the server will create a data link back from it's own port 20 to the port specified in the original connection by the client through a PORT command (non-passive connection, the most usual).

However, the client can instead of sending a PORT command, send a PASV command, which means the server is to send a message back to the client with a listening port number, so the client can itself create the data link (passive connection).

This gives in all 4 different combinations to take into account when configuring a port oriented firewall:

1) FTP server doing non-passive connection.
2) FTP server doing passive connection.
3) FTP client doing non-passive connection.
4) FTP client doing passive connection.

The easy way is to simply allow FTP servers and clients unlimited access through your firewall, but if you like to limit their access to "known" ports, you have to understand the 4 different scenarios.

1) The FTP server should be allowed to accept TCP connections to port 21, and to make TCP connections from port 20 to any (remote ephemeral) port.

2) The FTP server should be allowed to accept TCP connections to port 21, AND to accept TCP connections to any ephemeral port as well!

3) The FTP client should be allowed to make TCP connections to port 21, and to accept TCP connections from port 20 to any ephemeral port.

4) The FTP client should be allowed to make TCP connections to port 21, and to make TCP connections to any other (remote ephemeral) port as well!

Schematically:

Non Passive

Passive

An ephemeral (or dynamic) port is a temporary, short lived port, assigned to the application by the IP stack, taken from a specified pool. Mike Gleason has a wonderful page describing this principle in relation to FTP and firewalls as well. The range of ports used as ephemeral ports is both rather large, and can vary from system to system, both adding to the complexity of configuring a firewall correctly. Whenever an application makes a TCP connection, it is (usually) assigned a port by the IP stack as a "return address", meaning a TCP connection can have any port as its origin, and that is not a problem for firewalls at all, as they mostly check the destination port to figure out what to do. When FTP is involved, ephemeral ports are used as destination ports too, and that complicates the matter drastically.

Server Side

If you have an FTP server, you should configure both case 1) and 2). 1) is easy enough, it is case 2) that needs consideration. It should be obvious that you trust your FTP server not to do any hanky panky, so allowing it to accept TCP connections to any ephemeral port should NOT be a problem. (In windows, the ephemeral port range is by default 1024-5000, but the upper range can be changed.)
Sometimes, the FTP server can itself be assigned a range to use for PASV connections, as is the case with Cerberus that I am using. This makes it possible to define a range of ports that are not too difficult to configure in the firewall without any problems. When configuring this range, you must be carefull not to choose too narrow a range. First of all, several ports can be used by the same connection, second, you should be able to handle several clients at the same time, and third, an ephemeral port has a rather long latency period (minutes), before Windows allows that port to be reused, so you can very quickly run out of ports! Make sure you select a port range of at least several hundred ports. If you are having problems downloading and uploading to the server, but not connecting, this range is the first thing to check, if you are sure your firewall is configured correctly.

If your server is behind a router, you will have to configure the router as well. Note that routing port 21 is not enough, the chosen range of ports for PASV connections must also be routed correctly. Of course, all outgoing connections must be possible as well.

Client Side

You probably also want to be able to act as an FTP client. This time, both case 3) and 4) pose problems: In case 3), you want to allow any incoming connection to your FTP client from port 20. This sounds acceptable, but in some cases could be very dangerous, as we shall see in a moment. In case 4), the FTP client should be allowed to do connections to any port (returned from an FTP server), and this also sounds acceptable, but again, it may not be, as we shall see.

There are 3 basic cases, that need to be configured.
a) You use a simple FTP client program.
b) You use a browser like Internet Explorer as FTP client.
c) You connect through a proxy server running ProxyPlus, for instance, like I am myself.

In case a), it is not a problem to allow it to accept connections on any port from port 20. There is of course a risc of a hacking attempt from port 20, but it requires a very lucky punch to hit your port while it is active, and even more luck to actually fool the FTP program, so I consider that risc remote indeed. But you also need to allow the client to make TCP connections to any other port. Again, we must assume you trust your FTP client, so this should not be a problem either.

Case b), and the trouble begins... Again, the chance of someone finding an open port waiting for connections from port 20 is remote (case 3), but do you trust the Internet Explorer or any other browser to allow it to connect to just any port (case 4)? Especially in these days, when browsers are running ActiveX components and scripts of all kinds. You might argue, that any malicious script or component would of course choose to report back on port 80, as this port is of course always open for the browser to connect to. Still, I don't like the idea, so think about it before allowing your browser to do passive connection. (In Internet Explorer 6 SP1, this can be configured in the Options dialog, Advanced, Browsing.) Lucky for me, I am using ProxyPlus as a proxy, so my browser cannot connect to just any machine, and it is not a big problem with Kerio and Tiny personal firewall to configure it to ask for accept on new connection types. That way I can accept the connection explicitly, whenever I need to do FTP, which is not often, without giving the firewall any definite rule.

Case c) is the really difficult one! The proxy server need to be acting as an FTP server to the clients, but that is not a big problem as clients have known addresses, so these addresses can just be given the required permissions. But on the outside, the job is not all that simple.
Case 3) requires it to accept connections on any port from port 20, but as my proxy server has a lot of other ports open for other cases, the chance of a hacking attempt against it from a port 20 is much higher, opposed to the case of the client, which only had "stupid" ports open, waiting for connections from port 20. The ProxyPlus application is also offering a LOT of other services that could be abused. Of course, I have configured my proxy to only perform services when requested from "secure" clients, but I don't like the idea of just opening all ports! Also, only the ephemeral port range is relevant, but that range does indeed include ports actually used for some of the proxy server's actual services.

One way to do it is to create rules DISALLOWING connections from the outside to these sensitive ports BEFORE the rule allowing connections to any ephemeral port. Another way is to reconfigure the proxy server's service ports to fall outside the ephemeral port range, so the dangerous rule allowing connections does not interfere with the real service ports.

If you are behind a router, you can choose to route the entire ephemeral port range back to your client machine, or you can choose to do PASV FTP connections, which does not require any odd incoming ports open, but only outgoing connections.

I hope this discussion has given you enough information to decide for yourself how to do it.

Getting started with iptables in Linux

2012-04-18T02:24:00.002-07:00

Linux machines are known to be pretty secure. Studies have shown that Linux has been designed in a secure manner. Yet, despite all the security features that come bundled with a Linux installation, you need to configure these features correctly to make them work for you. I’ll guide you through the process of setting up of one of the tools that help secure your machine – the firewall. We will use the iptables firewall for this exercise. I am assuming that you are using a server running Red Hat Enterprise Linux 4 or similar. However, most of the steps should work fine on other Linux distributions as well. In this article we will setup a firewall on a Linux server running the Apache Web Server, FTP, and SSH.

Let us first see what ports these applications use and which of them need to have a port open on the firewall.

The Apache web server runs on port 80 by default. Apache is going to server all our web content on this port, therefore we need to keep this port open on the firewall. The SSH service runs on port 22. We need to be able to remotely connect to our server to work, so we keep it open. FTP runs on port 21 and it too needs the port to be open to communication.

Next, make sure you have iptables installed. Run this command as the root user:

# rpm -qa | grep iptables

If you have iptables installed the system should give you the version of iptables you have installed. In case you don’t you can try something like the following to get it and start it:

# yum install iptables

# /etc/init.d/iptables start

To check what kind of configuration iptables is currently running with:

# iptables –list
Chain INPUT (policy ACCEPT)
target prot opt source destination

Chain FORWARD (policy ACCEPT)
target prot opt source destination

Chain OUTPUT (policy ACCEPT)
target prot opt source destination

This command will list out all the firewall rules that have been set currently. I will proceed with the assumption that you do not have any firewall rules in your iptables configuration. Let’s now configure the firewall to allow open communication on the ports 80 for your web server, 22 for SSH, and port 21 for FTP. We’ll also make sure that we block communication to any port other than specified.

Here’s a firewall script configuration script. Create a new file and call it iptable-firewall.sh. Copy the following text into it:

#!/bin/sh
ANY=”0/0″
OPEN_PORTS=”21 22 80″
iptables -P INPUT ACCEPT
iptables -P FORWARD ACCEPT
iptables -P OUTPUT ACCEPT
# Flush (-F) all specific rules
iptables -F INPUT
iptables -F FORWARD
iptables -F OUTPUT
for port in $OPEN_PORTS
do
iptables -A INPUT -i eth0 -p tcp -s $ANY -d $ANY –destination-port $port –syn -j ACCEPT
iptables -A INPUT -i eth1 -p tcp -s $ANY -d $ANY –destination-port $port –syn -j ACCEPT
done
iptables -A INPUT -i eth1 -p icmp -s $ANY -d $ANY -j ACCEPT
#Allow any related/established connections
iptables -A INPUT -i eth0 -m state –state ESTABLISHED,RELATED -j ACCEPT
iptables -A INPUT -i eth1 -m state –state ESTABLISHED,RELATED -j ACCEPT
#Kill everything else
iptables -A INPUT -i eth0 -j DROP
iptables -A INPUT -i eth1 -j DROP
#write for boot
iptables-save > /etc/sysconfig/iptables

Now save the above file, grant it executable permissions and then run it:

# chmod +x iptable-firewall.sh

# ./iptable-firewall.sh

Now check your firewall rules:

# iptables –list

All your firewall rules should now be set. Your server is now secure. To make any modification or additions to this set of rules, edit the line where the OPEN_PORTSparameter is defined and add or remove ports form the list. Remember to run the script again after making any changes to it.

Linux Firewalls Using iptables

2012-04-18T00:18:00.002-07:00

Introduction

Network security is a primary consideration in any decision to host a website as the threats are becoming more widespread and persistent every day. One means of providing additional protection is to invest in a firewall. Though prices are always falling, in some cases you may be able to create a comparable unit using the Linux iptables package on an existing server for little or no additional expenditure.

This chapter shows how to convert a Linux server into:
A firewall while simultaneously being your home website's mail, web and DNS server.
A router that will use NAT and port forwarding to both protect your home network and have another web server on your home network while sharing the public IP address of your firewall.
Creating an iptables firewall script requires many steps, but with the aid of the sample tutorials, you should be able to complete a configuration relatively quickly.

What Is iptables?

Originally, the most popular firewall/NAT package running on Linux was ipchains, but it had a number of shortcomings. To rectify this, the Netfilter organization decided to create a new product called iptables, giving it such improvements as:

Better integration with the Linux kernel with the capability of loading iptables-specific kernel modules designed for improved speed and reliability.
Stateful packet inspection. This means that the firewall keeps track of each connection passing through it and in certain cases will view the contents of data flows in an attempt to anticipate the next action of certain protocols. This is an important feature in the support of active FTP and DNS, as well as many other network services.
Filtering packets based on a MAC address and the values of the flags in the TCP header. This is helpful in preventing attacks using malformed packets and in restricting access from locally attached servers to other networks in spite of their IP addresses.
System logging that provides the option of adjusting the level of detail of the reporting.
Better network address translation.
Support for transparent integration with such Web proxy programs as Squid.
A rate limiting feature that helps iptables block some types of denial of service (DoS) attacks.

Considered a faster and more secure alternative to ipchains, iptables has become the default firewall package installed under RedHat and Fedora Linux.

Download And Install The Iptables Package

Before you begin, you need to make sure that the iptables software RPM is installed. (See Chapter 6, "Installing Linux Software", if you need a refresher.) When searching for the RPMs, remember that the filename usually starts with the software package name by a version number, as in iptables-1.2.9-1.0.i386.rpm.

How To Start iptables

You can start, stop, and restart iptables after booting by using the commands:

[root@bigboy tmp]# service iptables start
[root@bigboy tmp]# service iptables stop
[root@bigboy tmp]# service iptables restart

To get iptables configured to start at boot, use the chkconfig command:.

[root@bigboy tmp]# chkconfig iptables on

Determining The Status of iptables

You can determine whether iptables is running or not via the service iptables status command. Fedora Core will give a simple status message. For example

[root@bigboy tmp]# service iptables status
Firewall is stopped.
[root@bigboy tmp]#

Packet Processing In iptables

All packets inspected by iptables pass through a sequence of built-in tables (queues) for processing. Each of these queues is dedicated to a particular type of packet activity and is controlled by an associated packet transformation/filtering chain.

There are three tables in total. The first is the mangle table which is responsible for the alteration of quality of service bits in the TCP header. This is hardly used in a home or SOHO environment.

The second table is the filter queue which is responsible for packet filtering. It has three built-in chains in which you can place your firewall policy rules. These are the:

Forward chain: Filters packets to servers protected by the firewall.
Input chain: Filters packets destined for the firewall.
Output chain: Filters packets originating from the firewall.

The third table is the nat queue which is responsible for network address translation. It has two built-in chains; these are:

Pre-routing chain: NATs packets when the destination address of the packet needs to be changed.
Post-routing chain: NATs packets when the source address of the packet needs to be changed

Table 14-1 Processing For Packets Routed By The Firewall

Queue Type	Queue Function	Packet Transformation Chain in Queue	Chain Function
Filter	Packet filtering	FORWARD	Filters packets to servers accessible by another NIC on the firewall.
		INPUT	Filters packets destined to the firewall.
		OUTPUT	Filters packets originating from the firewall
Nat	Network Address Translation	PREROUTING	Address translation occurs before routing. Facilitates the transformation of the destination IP address to be compatible with the firewall's routing table. Used with NAT of the destination IP address, also known as destination NAT or DNAT.
		POSTROUTING	Address translation occurs after routing. This implies that there was no need to modify the destination IP address of the packet as in pre-routing. Used with NAT of the source IP address using either one-to-one or many-to-one NAT. This is known as source NAT, or SNAT.
		OUTPUT	Network address translation for packets generated by the firewall. (Rarely used in SOHO environments)
Mangle	TCP header modification	PREROUTING POSTROUTING OUTPUT INPUT FORWARD	Modification of the TCP packet quality of service bits before routing occurs. (Rarely used in SOHO environments)

You need to specify the table and the chain for each firewall rule you create. There is an exception: Most rules are related to filtering, so iptables assumes that any chain that's defined without an associated table will be a part of the filter table. The filter table is therefore the default.

To help make this clearer, take a look at the way packets are handled by iptables. In Figure 14.1 a TCP packet from the Internet arrives at the firewall's interface on Network A to create a data connection.

The packet is first examined by your rules in the mangle table's PREROUTING chain, if any. It is then inspected by the rules in the nat table's PREROUTING chain to see whether the packet requires DNAT. It is then routed.

If the packet is destined for a protected network, then it is filtered by the rules in the FORWARD chain of the filter table and, if necessary, the packet undergoes SNAT in the POSTROUTING chain before arriving at Network B. When the destination server decides to reply, the packet undergoes the same sequence of steps. Both the FORWARD and POSTROUTING chains may be configured to implement quality of service (QoS) features in their mangle tables, but this is not usually done in SOHO environments.

If the packet is destined for the firewall itself, then it passes through the mangle table of the INPUT chain, if configured, before being filtered by the rules in the INPUT chain of the filter table before. If it successfully passes these tests then it is processed by the intended application on the firewall.

At some point, the firewall needs to reply. This reply is routed and inspected by the rules in the OUTPUT chain of the mangle table, if any. Next, the rules in the OUTPUT chain of the nat table determine whether DNAT is required and the rules in the OUTPUT chain of the filter table are then inspected to help restrict unauthorized packets. Finally, before the packet is sent back to the Internet, SNAT and QoS mangling is done by the POSTROUTING chain

Figure 14-1 Iptables Packet Flow Diagram

It is now time to discuss the ways in which you add rules to these chains.

Targets And Jumps

Each firewall rule inspects each IP packet and then tries to identify it as the target of some sort of operation. Once a target is identified, the packet needs to jump over to it for further processing. Table 14.2 lists the built-in targets that iptables uses.

Table 14-2 Descriptions Of The Most Commonly Used Targets

target	Desciption	Most Common Options
`ACCEPT`	iptables stops further processing. The packet is handed over to the end application or the operating system for processing	N/A
`DROP`	iptables stops further processing. The packet is blocked	N/A
`LOG`	The packet information is sent to the syslog daemon for logging iptables continues processing with the next rule in the table As you can't log and drop at the same time, it is common to have two similar rules in sequence. The first will log the packet, the second will drop it.	--log-prefix "string" Tells iptables to prefix all log messages with a user defined string. Frequently used to tell why the logged packet was dropped
`REJECT`	Works like the DROP target, but will also return an error message to the host sending the packet that the packet was blocked	--reject-with qualifier The qualifier tells what type of reject message is returned. Qualifiers include: icmp-port-unreachable (default) icmp-net-unreachable icmp-host-unreachable icmp-proto-unreachable icmp-net-prohibited icmp-host-prohibited tcp-reset echo-reply
`DNAT`	Used to do destination network address translation. ie. rewriting the destination IP address of the packet	--to-destination ipaddress Tells iptables what the destination IP address should be
`SNAT`	Used to do source network address translation rewriting the source IP address of the packet The source IP address is user defined	--to-source <address>[-<address>][:<port>-<port>] Specifies the source IP address and ports to be used by SNAT.
`MASQUERADE`	Used to do Source Network Address Translation. By default the source IP address is the same as that used by the firewall's interface	[--to-ports <port>[-<port>]] Specifies the range of source ports to which the original source port can be mapped.

Important Iptables Command Switch Operations

Each line of an iptables script not only has a jump, but they also have a number of command line options that are used to append rules to chains that match your defined packet characteristics, such the source IP address and TCP port. There are also options that can be used to just clear a chain so you can start all over again. Tables 14.2 through 14.6 list the most common options.

Table 14-2 General Iptables Match Criteria

iptables command Switch	Desciption
`-t <-table->`	If you don't specify a table, then the `filter` table is assumed. As discussed before, the possible built-in tables include: filter, nat, mangle
`-j <target>`	Jump to the specified target chain when the packet matches the current rule.
`-A`	Append rule to end of a chain
`-F`	Flush. Deletes all the rules in the selected table
`-p <protocol-type>`	Match protocol. Types include, icmp, tcp, udp, and all
`-s <ip-address>`	Match source IP address
`-d <ip-address>`	Match destination IP address
`-i <interface-name>`	Match "input" interface on which the packet enters.
`-o <interface-name>`	Match "output" interface on which the packet exits

In this command switches example

iptables -A INPUT -s 0/0 -i eth0 -d 192.168.1.1  -p TCP -j ACCEPT

iptables is being configured to allow the firewall to accept TCP packets coming in on interface eth0 from any IP address destined for the firewall's IP address of 192.168.1.1. The 0/0 representation of an IP address means any.

Table 14-4 Common TCP and UDP Match Criteria

Switch	Desciption
`-p tcp --sport <port>`	TCP source port. Can be a single value or a range in the format: start-port-number:end-port-number
`-p tcp --dport <port>`	TCP destination port. Can be a single value or a range in the format: starting-port:ending-port
`-p tcp --syn`	Used to identify a new TCP connection request. ! --syn means, not a new connection request
`-p udp --sport <port>`	UDP source port. Can be a single value or a range in the format: starting-port:ending-port
`-p udp --dport <port>`	UDP destination port. Can be a single value or a range in the format: starting-port:ending-port

In this example:

iptables -A FORWARD -s 0/0 -i eth0 -d 192.168.1.58 -o eth1 -p TCP \
         --sport 1024:65535 --dport 80 -j ACCEPT

iptables is being configured to allow the firewall to accept TCP packets for routing when they enter on interface eth0 from any IP address and are destined for an IP address of 192.168.1.58 that is reachable via interface eth1. The source port is in the range 1024 to 65535 and the destination port is port 80 (www/http).

Table 14-5 Common ICMP (Ping) Match Criteria

Matches used with ---icmp-type	Desciption
`--icmp-type <type>`	The most commonly used types are echo-reply and echo-request

In this example:

iptables -A OUTPUT -p icmp --icmp-type echo-request -j ACCEPT
iptables -A INPUT  -p icmp --icmp-type echo-reply   -j ACCEPT

iptables is being configured to allow the firewall to send ICMP echo-requests (pings) and in turn, accept the expected ICMP echo-replies.

Consider another example

iptables -A INPUT -p icmp --icmp-type echo-request \
         -m limit --limit 1/s -i eth0 -j ACCEPT

The limit feature in iptables specifies the maximum average number of matches to allow per second. You can specify time intervals in the format /second, /minute, /hour, or /day, or you can use abbreviations so that 3/second is the same as 3/s.

In this example, ICMP echo requests are restricted to no more than one per second. When tuned correctly, this feature allows you to filter unusually high volumes of traffic that characterize denial of service (DOS) attacks and Internet worms.

iptables -A INPUT -p tcp --syn -m limit --limit 5/s -i eth0 -j ACCEPT

You can expand on the limit feature of iptables to reduce your vulnerability to certain types of denial of service attack. Here a defense for SYN flood attacks was created by limiting the acceptance of TCP segments with the SYN bit set to no more than five per second.

Table 14-6 Common Extended Match Criteria

Switch	Desciption
`-m multiport --sports <port, port>`	A variety of TCP/UDP source ports separated by commas. Unlike when `-m` isn't used, they do not have to be within a range.
`-m multiport --dports <port, port>`	A variety of TCP/UDP destination ports separated by commas. Unlike when `-m` isn't used, they do not have to be within a range.
`-m multiport --ports <port, port>`	A variety of TCP/UDP ports separated by commas. Source and destination ports are assumed to be the same and they do not have to be within a range.
`-m --state <state>`	The most frequently tested states are: ESTABLISHED: The packet is part of a connection that has seen packets in both directions NEW: The packet is the start of a new connection RELATED: The packet is starting a new secondary connection. This is a common feature of such protocols such as an FTP data transfer, or an ICMP error. INVALID: The packet couldn't be identified. Could be due to insufficient system resources, or ICMP errors that don't match an existing data flow.

This is an expansion on the previous example:

iptables -A FORWARD -s 0/0 -i eth0 -d 192.168.1.58 -o eth1 -p TCP \
         --sport 1024:65535 -m multiport --dports 80,443 -j ACCEPT
 
iptables -A FORWARD -d 0/0 -o eth0 -s 192.168.1.58 -i eth1 -p TCP \
         -m state --state ESTABLISHED -j ACCEPT

Here iptables is being configured to allow the firewall to accept TCP packets to be routed when they enter on interface eth0 from any IP address destined for IP address of 192.168.1.58 that is reachable via interface eth1. The source port is in the range 1024 to 65535 and the destination ports are port 80 (www/http) and 443 (https). The return packets from 192.168.1.58 are allowed to be accepted too. Instead of stating the source and destination ports, you can simply allow packets related to established connections using the -m state and --state ESTABLISHED options.

Using User Defined Chains

As you may remember, you can configure iptables to have user-defined chains. This feature is frequently used to help streamline the processing of packets. For example, instead of using a single, built-in chain for all protocols, you can use the chain to determine the protocol type for the packet and then hand off the actual final processing to a user-defined, protocol-specific chain in the filter table. In other words, you can replace a long chain with a stubby main chain pointing to multiple stubby chains, thereby shortening the total length of all chains the packet has to pass through. For example

iptables -A INPUT -i eth0  -d 206.229.110.2 -j fast-input-queue
iptables -A OUTPUT -o eth0 -s 206.229.110.2 -j fast-output-queue

iptables -A fast-input-queue  -p icmp -j icmp-queue-in
iptables -A fast-output-queue -p icmp -j icmp-queue-out

iptables -A icmp-queue-out -p icmp --icmp-type echo-request \
         -m state --state NEW -j ACCEPT

iptables -A icmp-queue-in -p icmp --icmp-type echo-reply -j ACCEPT

Here six queues help assist in improving processing speed. Table 14.7 summarizes the function of each.

Table 14.7 Custom Queues Example Listing

Chain	Desciption
`INPUT`	The regular built-in INPUT chain in iptables
`OUTPUT`	The regular built-in OUTPUT chain in `iptables`
`fast-input-queue`	Input chain dedicated to identifying specific protocols and shunting the packets to protocol specific chains.
`fast-output-queue`	Output chain dedicated to identifying specific protocols and shunting the packets to protocol specific chains.
`icmp-queue-out`	Output queue dedicated to ICMP
`icmp-queue-in`	Input queue dedicated to ICMP

Saving Your iptables Scripts

The service iptables save command permanently saves the iptables configuration in the /etc/sysconfig/iptables file. When the system reboots, the iptables-restore program reads the configuration and makes it the active configuration.

The format of the /etc/sysconfig/iptables file is slightly different from that of the scripts shown in this chapter. The initialization of built-in chains is automatic and the string "iptables" is omitted from the rule statements.

Here is a sample /etc/sysconfig/iptables configuration that allows ICMP, IPSec (ESP and AH packets), already established connections, and inbound SSH.

[root@bigboy tmp]# cat /etc/sysconfig/iptables
# Generated by iptables-save v1.2.9 on Mon Nov 8 11:00:07 2004 
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [144:12748]
:RH-Firewall-1-INPUT - [0:0]
-A INPUT -j RH-Firewall-1-INPUT
-A FORWARD -j RH-Firewall-1-INPUT
-A RH-Firewall-1-INPUT -i lo -j ACCEPT
-A RH-Firewall-1-INPUT -p icmp -m icmp --icmp-type 255 -j ACCEPT
-A RH-Firewall-1-INPUT -p esp -j ACCEPT
-A RH-Firewall-1-INPUT -p ah -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A RH-Firewall-1-INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT
-A RH-Firewall-1-INPUT -j REJECT --reject-with icmp-host-prohibited
COMMIT
# Completed on Mon Nov 8 11:00:07 2004
[root@bigboy tmp]#

It is never a good idea to edit this script directly because it is always overwritten by the save command and it doesn't save any comments at all, which can also make it extremely difficult to follow. For these reasons, you're better off writing and applying a customized script and then using the service iptables save command to make the changes permanent.

Fedora's iptables Rule Generator

Fedora comes with a program called lokkit that you can use to generate a very rudimentary firewall rule set. It prompts for the level of security and then gives you the option of doing simple customizations. It is a good place for beginners to start on a test system so that they can see a general rule structure.

Like the service iptables save command, lokkit saves the firewall rules in a new /etc/sysconfig/iptables file for use on the next reboot.

Once you have become familiar with the iptables syntax, it's best to write scripts that you can comment and then save it to /etc/sysconfig/iptables. It makes them much more manageable and readable.

Recovering From A Lost Script

Sometimes the script you created to generate iptables rules may get corrupted or lost, or you might inherit a new system from an administer and cannot find the original script used to protect it. In these situations, you can use the iptables-save and iptables-restore commands to assist you with the continued management of the server.

Unlike the service iptables save command, which actually saves a permanent copy of the firewall's active configuration in the /etc/sysconfig/iptables file, iptables-save displays the active configuration to the screen in /etc/sysconfig/iptables format. If you redirect the iptables-save screen output to a file with the > symbol, then you can edit the output and reload the updated rules when they meet your new criteria with the iptables-restore command.

This example exports the iptables-save output to a text file named firewall-config.

[root@bigboy tmp]# iptables-save > firewall-config
[root@bigboy tmp]# cat firewall-config
# Generated by iptables-save v1.2.9 on Mon Nov 8 11:00:07 2004
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [144:12748]
:RH-Firewall-1-INPUT - [0:0]
-A INPUT -j RH-Firewall-1-INPUT
-A FORWARD -j RH-Firewall-1-INPUT
-A RH-Firewall-1-INPUT -i lo -j ACCEPT
-A RH-Firewall-1-INPUT -p icmp -m icmp --icmp-type 255 -j ACCEPT
-A RH-Firewall-1-INPUT -p esp -j ACCEPT
-A RH-Firewall-1-INPUT -p ah -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A RH-Firewall-1-INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT
-A RH-Firewall-1-INPUT -j REJECT --reject-with icmp-host-prohibited
COMMIT
# Completed on Mon Nov 8 11:00:07 2004
[root@bigboy tmp]#

After editing the firewall-config file with the commands you need, you can reload it into the active firewall rule set with the iptables-restore command.

[root@bigboy tmp]# iptables-restore < firewall-config

Finally, you should permanently save the active configuration so that it will be loaded automatically when the system reboots:

[root@bigboy tmp]# service iptables save

If desired, you can eventually convert this firewall-config file into a regular iptables script so that it becomes more easily recognizable and manageable.

Loading Kernel Modules Needed By iptables

The iptables application requires you to load certain kernel modules to activate some of its functions. Whenever any type of NAT is required, the iptable_nat module needs to be loaded. The ip_conntrack_ftp module needs to be added for FTP support and should always be loaded with the ip_conntrack module which tracks TCP connection states. As most scripts probably will keep track of connection states, the ip_conntrack module will be needed in any case. The ip_nat_ftp module also needs to be loaded for FTP servers behind a NAT firewall.

Unfortunately, the /etc/sysconfig/iptables file doesn't support the loading of modules, so you'll have to add the statements to your /etc/rc.local file which is run at the end of every reboot.

The script samples in this chapter include these statements only as a reminder to place them in the /etc/rc.local file

# File: /etc/rc.local

# Module to track the state of connections
modprobe ip_conntrack

# Load the iptables active FTP module, requires ip_conntrack
modprobe ip_conntrack_ftp
  
# Load iptables NAT module when required
modprobe iptable_nat
 
# Module required for active an FTP server using NAT
modprobe ip_nat_ftp

Sample iptables Scripts

This section provides some sample scripts you can use to get iptables working for you. Pay special attention to the logging example at the end.

The basic initialization script snippet should also be included in all your scripts to ensure the correct initialization of your chains should you decide to restart your script after startup. This chapter also includes other snippets that will help you get basic functionality. It should be a good guide to get you started.

Note: Once you feel more confident, you can use Appendix II "Codes, Scripts, and Configurations", to find detailed scripts. The appendix shows you how to allow your firewall to:

Be used as a Linux Web, mail and DNS server
Be the NAT router for your home network
Prevent various types of attacks using corrupted TCP, UDP and ICMP packets.
Provide outbound passive FTP access from the firewall

There are also simpler code snippets in the Appendix II "Codes, Scripts, and Configurations", for Inbound and outbound FTP connections to and from your firewall

Basic Operating System Defense

You can do several things before employing your firewall script to improve the resilience of your firewall to attack. For example, the Linux operating system has a number of built-in protection mechanisms that you should activate by modifying the system kernel parameters in the /proc filesystem via the /etc/sysctl.conf file. Using of /etc/sysctl.conf to modify kernel parameters is explained in more detail in , Appendix I "Miscellaneous Linux Topics".

Here is a sample configuration:

# File: /etc/sysctl.conf
 
#---------------------------------------------------------------
# Disable routing triangulation. Respond to queries out
# the same interface, not another. Helps to maintain state
# Also protects against IP spoofing
#---------------------------------------------------------------
 
net/ipv4/conf/all/rp_filter = 1
 
 
#---------------------------------------------------------------
# Enable logging of packets with malformed IP addresses
#---------------------------------------------------------------
 
net/ipv4/conf/all/log_martians = 1
 
 
#---------------------------------------------------------------
# Disable redirects
#---------------------------------------------------------------
 
net/ipv4/conf/all/send_redirects = 0
 
 
#---------------------------------------------------------------
# Disable source routed packets
#---------------------------------------------------------------
 
net/ipv4/conf/all/accept_source_route = 0
 
 
#---------------------------------------------------------------
# Disable acceptance of ICMP redirects
#---------------------------------------------------------------
 
net/ipv4/conf/all/accept_redirects = 0
 
 
#---------------------------------------------------------------
# Turn on protection from Denial of Service (DOS) attacks
#---------------------------------------------------------------
 
net/ipv4/tcp_syncookies = 1
 
 
#---------------------------------------------------------------
# Disable responding to ping broadcasts
#---------------------------------------------------------------
 
net/ipv4/icmp_echo_ignore_broadcasts = 1
 
#---------------------------------------------------------------
# Enable IP routing. Required if your firewall is protecting a
# network, NAT included
#---------------------------------------------------------------

net/ipv4/ip_forward = 1

Advanced iptables Initialization

You may also want to add some more advanced initialization steps to your script, including checks for Internet traffic from RFC1918 private addresses. The sample script snippet below outlines how to do this. More complex initializations would include checks for attacks using invalid TCP flags and directed broadcasts which are beyond the scope of this book.

The script also uses multiple user-defined chains to make the script shorter and faster as the chains can be repeatedly accessed. This removes the need to repeat the same statements over and over again.

You can take even more precautions to further protect your network. The complete firewall script in Appendix II "Codes, Scripts, and Configurations", outlines most of them.

#=#=#=#=#=#=#=#=#=#=#=#=#=#=#=#=#=#=#=#=#=#=#=#=#=#=#=#=#=#=#=#=
#
# Define networks: NOTE!! You may want to put these "EXTERNAL"
# definitions at the top of your script.
#
#=#=#=#=#=#=#=#=#=#=#=#=#=#=#=#=#=#=#=#=#=#=#=#=#=#=#=#=#=#=#=#=
 
EXTERNAL_INT="eth0"            # External Internet interface
EXTERNAL_IP="97.158.253.25"    # Internet Interface IP address
 
#---------------------------------------------------------------
# Initialize our user-defined chains
#---------------------------------------------------------------
 
iptables -N valid-src
iptables -N valid-dst
 
#---------------------------------------------------------------
# Verify valid source and destination addresses for all packets
#---------------------------------------------------------------
 
iptables -A INPUT   -i $EXTERNAL_INT -j valid-src
iptables -A FORWARD -i $EXTERNAL_INT -j valid-src
iptables -A OUTPUT  -o $EXTERNAL_INT -j valid-dst
iptables -A FORWARD -o $EXTERNAL_INT -j valid-dst
 
#=#=#=#=#=#=#=#=#=#=#=#=#=#=#=#=#=#=#=#=#=#=#=#=#=#=#=#=#=#=#=#
#
# Source and Destination Address Sanity Checks
#
# Drop packets from networks covered in RFC 1918 (private nets)
# Drop packets from external interface IP
#
#=#=#=#=#=#=#=#=#=#=#=#=#=#=#=#=#=#=#=#=#=#=#=#=#=#=#=#=#=#=#=#
 
iptables -A valid-src -s 10.0.0.0/8     -j DROP
iptables -A valid-src -s 172.16.0.0/12  -j DROP
iptables -A valid-src -s 192.168.0.0/16 -j DROP
iptables -A valid-src -s 224.0.0.0/4    -j DROP
iptables -A valid-src -s 240.0.0.0/5    -j DROP
iptables -A valid-src -s 127.0.0.0/8    -j DROP
iptables -A valid-src -s 0.0.0.0/8       -j DROP
iptables -A valid-src -d 255.255.255.255 -j DROP
iptables -A valid-src -s 169.254.0.0/16  -j DROP
iptables -A valid-src -s $EXTERNAL_IP    -j DROP
iptables -A valid-dst -d 224.0.0.0/4    -j DROP

Allowing DNS Access To Your Firewall

You'll almost certainly want your firewall to make DNS queries to the Internet. This is not because it is required for the basic functionality of the firewall, but because of Fedora Linux's yum RPM updater which will help to keep the server up to date with the latest security patches. The following statements will apply not only for firewalls acting as DNS clients but also for firewalls working in a caching or regular DNS server role.

#---------------------------------------------------------------
# Allow outbound DNS queries from the FW and the replies too
#
# - Interface eth0 is the internet interface
#
# Zone transfers use TCP and not UDP. Most home networks
# / websites using a single DNS server won't require TCP statements
#
#---------------------------------------------------------------
 
iptables -A OUTPUT -p udp -o eth0 --dport 53 --sport 1024:65535 \
         -j ACCEPT
 
iptables -A INPUT -p udp -i eth0 --sport 53 --dport 1024:65535 \
         -j ACCEPT

Allowing WWW And SSH Access To Your Firewall

This sample snippet is for a firewall that doubles as a web server that is managed remotely by its system administrator via secure shell (SSH) sessions. Inbound packets destined for ports 80 and 22 are allowed thereby making the first steps in establishing a connection. It isn't necessary to specify these ports for the return leg as outbound packets for all established connections are allowed. Connections initiated by persons logged into the Web server will be denied as outbound NEW connection packets aren't allowed.

#---------------------------------------------------------------
# Allow previously established connections
# - Interface eth0 is the internet interface
#---------------------------------------------------------------

iptables -A OUTPUT -o eth0 -m state --state ESTABLISHED,RELATED \
  -j ACCEPT

#---------------------------------------------------------------
# Allow port 80 (www) and 22 (SSH) connections to the firewall
#---------------------------------------------------------------

iptables -A INPUT -p tcp -i eth0 --dport 22 --sport 1024:65535 \
  -m state --state NEW -j ACCEPT
iptables -A INPUT -p tcp -i eth0 --dport 80 --sport 1024:65535 \
  -m state --state NEW -j ACCEPT

Allowing Your Firewall To Access The Internet

This iptables script enables a user on the firewall to use a Web browser to surf the Internet. HTTP traffic uses TCP port 80, and HTTPS uses port 443.

Note: HTTPS (secure HTTP) is used for credit card transactions frequently, as well as by RedHat Linux servers running up2date. FTP and HTTP are frequently used with yum.

#---------------------------------------------------------------
# Allow port 80 (www) and 443 (https) connections from the firewall
#---------------------------------------------------------------

iptables -A OUTPUT -j ACCEPT -m state \
  --state NEW,ESTABLISHED,RELATED -o eth0 -p tcp \
  -m multiport --dports 80,443 --sport 1024:65535
 
#---------------------------------------------------------------
# Allow previously established connections
# - Interface eth0 is the internet interface
#---------------------------------------------------------------

iptables -A INPUT -j ACCEPT -m state --state ESTABLISHED,RELATED  \
-i eth0 -p tcp

If you want all TCP traffic originating from the firewall to be accepted, then remove the line:

-m multiport --dports 80,443 --sport 1024:65535

Allow Your Home Network To Access The Firewall

In this example, eth1 is directly connected to a home network using IP addresses from the 192.168.1.0 network. All traffic between this network and the firewall is simplistically assumed to be trusted and allowed.

Further rules will be needed for the interface connected to the Internet to allow only specific ports, types of connections and possibly even remote servers to have access to your firewall and home network.

#---------------------------------------------------------------
# Allow all bidirectional traffic from your firewall to the
# protected network
# - Interface eth1 is the private network interface
#---------------------------------------------------------------

iptables -A INPUT   -j ACCEPT -p all -s 192.168.1.0/24 -i eth1
iptables -A OUTPUT  -j ACCEPT -p all -d 192.168.1.0/24 -o eth1

Masquerading (Many to One NAT)

As explained in Chapter 2, "Introduction to Networking", masquerading is another name for what many call many to one NAT. In other words, traffic from all devices on one or more protected networks will appear as if it originated from a single IP address on the Internet side of the firewall.

Note: The masquerade IP address always defaults to the IP address of the firewall's main interface. The advantage of this is that you never have to specify the NAT IP address. This makes it much easier to configure iptables NAT with DHCP.

You can configure many to one NAT to an IP alias, using the POSTROUTING and not the MASQUERADE statement. An example of this can be seen in the static NAT section that follows.

Keep in mind that iptables requires the iptables_nat module to be loaded with the modprobe command for the masquerade feature to work. Masquerading also depends on the Linux operating system being configured to support routing between the internet and private network interfaces of the firewall. This is done by enabling IP forwarding or routing by giving the file /proc/sys/net/ipv4/ip_forward the value 1 as opposed to the default disabled value of 0.

Once masquerading has been achieved using the POSTROUTING chain of the nat table, you will have to configure iptables to allow packets to flow between the two interfaces. To do this, use the FORWARD chain of the filter table. More specifically, packets related to NEW and ESTABLISHED connections will be allowed outbound to the Internet, but only packets related to ESTABLISHED connections will be allowed inbound. This helps to protect the home network from anyone trying to initiate connections from the Internet:

#---------------------------------------------------------------
# Load the NAT module
#
# Note: It is best to use the /etc/rc.local example in this
#       chapter. This value will not be retained in the
#       /etc/sysconfig/iptables file. Included only as a reminder.
#---------------------------------------------------------------
 
modprobe iptable_nat

#---------------------------------------------------------------
# Enable routing by modifying the ip_forward /proc filesystem file
#
# Note: It is best to use the /etc/sysctl.conf example in this
#       chapter. This value will not be retained in the
#       /etc/sysconfig/iptables file. Included only as a reminder.
#---------------------------------------------------------------
 
echo 1 > /proc/sys/net/ipv4/ip_forward
 
#---------------------------------------------------------------
# Allow masquerading
# - Interface eth0 is the internet interface
# - Interface eth1 is the private network interface
#---------------------------------------------------------------
 
iptables -A POSTROUTING -t nat -o eth0 -s 192.168.1.0/24 -d 0/0 \
         -j MASQUERADE
 
#---------------------------------------------------------------
# Prior to masquerading, the packets are routed via the filter
# table's FORWARD chain.
# Allowed outbound: New, established and related connections
# Allowed inbound : Established and related connections
#---------------------------------------------------------------
 
iptables -A FORWARD -t filter -o eth0 -m state \
         --state NEW,ESTABLISHED,RELATED -j ACCEPT
 
iptables -A FORWARD -t filter -i eth0 -m state \
         --state ESTABLISHED,RELATED -j ACCEPT

Note: If you configure your firewall to do masquerading, then if should be the used as the default gateway for all your servers on the network.

Port Forwarding Type NAT (DHCP DSL)

In many cases home users may get a single DHCP public IP address from their ISPs. If a Linux firewall is also your interface to the Internet and you want to host a Web site on one of the NAT protected home servers, then you will have to use port forwarding. Here the combination of the firewall's single IP address, the remote server's IP address, and the source/destination port of the traffic can be used to uniquely identify a traffic flow. All traffic that matches a particular combination of these factors may then be forwarded to a single server on the private network.

Port forwarding is handled by the PREROUTING chain of the nat table. As in masquerading, the iptables_nat module has to be loaded and routing has to be enabled for port forwarding to work. Routing too must be allowed in iptables with the FORWARD chain, this includes all NEW inbound connections from the Internet matching the port forwarding port plus all future packets related to the ESTABLISHED connection in both directions:

#---------------------------------------------------------------
# Load the NAT module
#
# Note: It is best to use the /etc/rc.local example in this
#       chapter. This value will not be retained in the
#       /etc/sysconfig/iptables file. Included only as a reminder.
#---------------------------------------------------------------
 
modprobe iptable_nat

#---------------------------------------------------------------
# Get the IP address of the Internet interface eth0 (linux only)
#
# You'll have to use a different expression to get the IP address
# for other operating systems which have a different ifconfig output
# or enter the IP address manually in the PREROUTING statement
#
# This is best when your firewall gets its IP address using DHCP.
# The external IP address could just be hard coded ("typed in
# normally")
#---------------------------------------------------------------
 
external_int="eth0"
external_ip="`ifconfig $external_int | grep 'inet addr' | \
                       awk '{print $2}' | sed -e 's/.*://'`"
#---------------------------------------------------------------
# Enable routing by modifying the ip_forward /proc filesystem file
#
# Note: It is best to use the /etc/sysctl.conf example in this
#       chapter. This value will not be retained in the
#       /etc/sysconfig/iptables file. Included only as a reminder.
#---------------------------------------------------------------
 
echo 1 > /proc/sys/net/ipv4/ip_forward
 
#---------------------------------------------------------------
# Allow port forwarding for traffic destined to port 80 of the
# firewall's IP address to be forwarded to port 8080 on server
# 192.168.1.200
#
# - Interface eth0 is the internet interface
# - Interface eth1 is the private network interface
#---------------------------------------------------------------
 
iptables -t nat -A PREROUTING -p tcp -i eth0 -d $external_ip \
     --dport 80 --sport 1024:65535 -j DNAT --to 192.168.1.200:8080
 
#---------------------------------------------------------------
# After DNAT, the packets are routed via the filter table's
# FORWARD chain.
# Connections on port 80 to the target machine on the private
# network must be allowed.
#---------------------------------------------------------------
 
iptables -A FORWARD -p tcp -i eth0 -o eth1 -d 192.168.1.200 \
    --dport 8080 --sport 1024:65535 -m state --state NEW -j ACCEPT
 
iptables -A FORWARD -t filter -o eth0 -m state \
         --state NEW,ESTABLISHED,RELATED -j ACCEPT
 
iptables -A FORWARD -t filter -i eth0 -m state \
         --state ESTABLISHED,RELATED -j ACCEPT

Static NAT

In this example, all traffic to a particular public IP address, not just to a particular port, is translated to a single server on the protected subnet. Because the firewall has more than one IP address, I can't recommend MASQUERADE; it will force masquerading as the IP address of the primary interface and not as any of the alias IP addresses the firewall may have. Instead, use SNAT to specify the alias IP address to be used for connections initiated by all other servers in the protected network.

Note: Although the nat table NATs all traffic to the target servers (192.168.1.100 to 102), only connections on ports 80,443 and 22 are allowed through by the FORWARD chain. Also notice how you have to specify a separate -m multiport option whenever you need to match multiple non-sequential ports for both source and destination.

In this example the firewall:

Uses one to one NAT to make the server 192.168.1.100 on your home network appear on the Internet as IP addresses 97.158.253.26.
Creates a many to one NAT for the 192.168.1.0 home network in which all the servers appear on the Internet as IP address 97.158.253.29. This is different from masquerading

You will have to create alias IP addresses for each of these Internet IPs for one to one NAT to work.

#---------------------------------------------------------------
# Load the NAT module
#
# Note: It is best to use the /etc/rc.local example in this
#       chapter. This value will not be retained in the
#       /etc/sysconfig/iptables file. Included only as a reminder.
#---------------------------------------------------------------
 
modprobe iptable_nat
 
#---------------------------------------------------------------
# Enable routing by modifying the ip_forward /proc filesystem file
#
# Note: It is best to use the /etc/sysctl.conf example in this
#       chapter. This value will not be retained in the
#       /etc/sysconfig/iptables file. Included only as a reminder.
#---------------------------------------------------------------
 
echo 1 > /proc/sys/net/ipv4/ip_forward
 
#---------------------------------------------------------------
# NAT ALL traffic:
###########
# REMEMBER to create aliases for all the internet IP addresses below
###########
#
# TO:             FROM:           MAP TO SERVER:
# 97.158.253.26    Anywhere        192.168.1.100 (1:1 NAT - Inbound)
# Anywhere        192.168.1.100   97.158.253.26 (1:1 NAT - Outbound)
# Anywhere        192.168.1.0/24  97.158.253.29 (FW IP)
#
# SNAT is used to NAT all other outbound connections initiated
# from the protected network to appear to come from
# IP address 97.158.253.29
#
# POSTROUTING:
#   NATs source IP addresses. Frequently used to NAT connections from
#   your home network to the Internet
#
# PREROUTING:
#   NATs destination IP addresses. Frequently used to NAT
#   connections from the Internet to your home network
#
# - Interface eth0 is the internet interface
# - Interface eth1 is the private network interface
#---------------------------------------------------------------
 
# PREROUTING statements for 1:1 NAT
# (Connections originating from the Internet)

iptables -t nat -A PREROUTING -d 97.158.253.26 -i eth0 \
        -j DNAT --to-destination 192.168.1.100

# POSTROUTING statements for 1:1 NAT
# (Connections originating from the home network servers)

iptables -t nat -A POSTROUTING -s 192.168.1.100 -o eth0 \
         -j SNAT --to-source 97.158.253.26
 
# POSTROUTING statements for Many:1 NAT
# (Connections originating from the entire home network)

iptables -t nat -A POSTROUTING -s 192.168.1.0/24 \
         -j SNAT -o eth0 --to-source 97.158.253.29

# Allow forwarding to each of the servers configured for 1:1 NAT
# (For connections originating from the Internet. Notice how you
# use the real IP addresses here)
 
iptables -A FORWARD -p tcp -i eth0 -o eth1 -d 192.168.1.100 \
    -m multiport --dports 80,443,22 \
    -m state --state NEW -j ACCEPT
 
 
# Allow forwarding for all New and Established SNAT connections
# originating on the home network AND already established
# DNAT connections
 
iptables -A FORWARD -t filter -o eth0 -m state \
         --state NEW,ESTABLISHED,RELATED -j ACCEPT
 
# Allow forwarding for all 1:1 NAT connections originating on
# the Internet that have already passed through the NEW forwarding
# statements above
 
iptables -A FORWARD -t filter -i eth0 -m state \
         --state ESTABLISHED,RELATED -j ACCEPT

Troubleshooting iptables

A number of tools are at your disposal for troubleshooting iptables firewall scripts. One of the best methods is to log all dropped packets to the /var/log/messages file.

Checking The Firewall Logs

You track packets passing through the iptables list of rules using the LOG target. You should be aware that the LOG target:

Logs all traffic that matches the iptables rule in which it is located.
Automatically writes an entry to the /var/log/messages file and then executes the next rule.

If you want to log only unwanted traffic, therefore, you have to add a matching rule with a DROP target immediately after the LOG rule. If you don't, you'll find yourself logging both desired and unwanted traffic with no way of discerning between the two, because by default iptables doesn't state why the packet was logged in its log message.

This example logs a summary of failed packets to the file /var/log/messages. You can use the contents of this file to determine which TCP/UDP ports you need to open to provide access to specific traffic that is currently stopped.

#---------------------------------------------------------------
# Log and drop all other packets to file /var/log/messages
# Without this we could be crawling around in the dark
#---------------------------------------------------------------
 
iptables -A OUTPUT -j LOG
iptables -A INPUT -j LOG
iptables -A FORWARD -j LOG
 
iptables -A OUTPUT -j DROP
iptables -A INPUT -j DROP
iptables -A FORWARD -j DROP

Here are some examples of the output of this file:

Firewall denies replies to DNS queries (UDP port 53) destined to server 192.168.1.102 on the home network.

Feb 23 20:33:50 bigboy kernel: IN=wlan0 OUT= MAC=00:06:25:09:69:80:00:a0:c5:e1:3e:88:08:00 SRC=192.42.93.30 DST=192.168.1.102 LEN=220 TOS=0x00 PREC=0x00 TTL=54 ID=30485 PROTO=UDP SPT=53 DPT=32820 LEN=200

Firewall denies Windows NetBIOS traffic (UDP port 138)

Feb 23 20:43:08 bigboy kernel: IN=wlan0 OUT= MAC=ff:ff:ff:ff:ff:ff:00:06:25:09:6a:b5:08:00 SRC=192.168.1.100 DST=192.168.1.255 LEN=241 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=UDP SPT=138 DPT=138 LEN=221

Firewall denies Network Time Protocol (NTP UDP port 123)

Feb 23 20:58:48 bigboy kernel: IN= OUT=wlan0 SRC=192.168.1.102 DST=207.200.81.113 LEN=76 TOS=0x10 PREC=0x00 TTL=64 ID=0 DF PROTO=UDP SPT=123 DPT=123 LEN=56

The traffic in all these examples isn't destined for the firewall; Therefore, you should check your INPUT, OUTPUT, FORWARD, and NAT related statements. If the firewall's IP address is involved, then you should focus on the INPUT and OUTPUT statements

If nothing shows up in the logs, then follow the steps in Chapter 4, "Simple Network Troubleshooting", to determine whether the data is reaching your firewall at all and, if it is not, the location on your network that could be causing the problem.

As a general rule, you won't be able to access the public NAT IP addresses from servers on your home network. Basic NAT testing requires you to ask a friend to try to connect to your home network from the Internet.

You can then use the logging output in /var/log/messages to make sure that the translations are occurring correctly and iptables isn't dropping the packets after translation occurs.

iptables Won't Start

The iptables startup script expects to find the /etc/sysconfig/iptables before it starts. If none exists, then symptoms include the firewall status always being stopped and the /etc/init.d/iptables script running without the typical [OK] or [FAILED] messages.

If you have just installed iptables and have never applied a policy, then you will face this problem. Unfortunately, running the service iptables save command before restarting won't help either. You have to create this file.

[root@bigboy tmp]# service iptables start
[root@bigboy tmp]#

[root@bigboy tmp]# touch /etc/sysconfig/iptables
[root@bigboy tmp]# chmod 600 /etc/sysconfig/iptables

[root@bigboy tmp]# service iptables start
Applying iptables firewall rules: [  OK  ]
[root@bigboy tmp]#

Conclusion

A firewall is a critical part of any establishment that connects to an unprotected network such as the Internet, but a firewall is never sufficient. Web site security involves not just protection from corrupted packets or maliciously overwhelming volumes of traffic, but also involves daily data backups to help recovery from device failures, regular application patching, enforced password policies, restricted and monitored physical access to your servers, reliable power and cooling, secured cabling, redundant hardware, and, probably most importantly, well trained and motivated employees. Security should be viewed as anything that contributes to the desired risk-free functioning of your site, and it is well worth the money to invest in and learn from a book that specializes in the topic.

Truth might surprise U

2012-04-12T02:33:00.002-07:00

A 24 year old boy seeing out from the train's window shouted...
"Dad, look the trees are going behind!"
Dad smiled and a young couple sitting nearby, looked at the 24 year old's childish behavior with pity, suddenly he again exclaimed...
"Dad, look the clouds are running with us!"
The couple couldn't resist and said to the old man...
"Why don't you take your son to a good doctor?"
The old man smiled and said...
"I did and we are just coming from the hospital, my son was blind from birth, he just got his eyes today."
MORAL..........
Every single person on the planet has a story. Don't judge people before you truly know them

Getting Started with J2SSH

2012-04-12T02:31:00.005-07:00

The first thing you will probably want to do is to connect to an SSH server using J2SSH. This is a fairly
straightforward procedure using the SshClient class. This class provides access for connecting, authenticating
and starting a session channel, which enables you to execute commands or start the users shell.

import com.sshtools.j2ssh.SshClient;

First of all prepare your application, in this section we will guide you through the basics so for now just a simple
try/catch inside the static main method.

import java.io.BufferedReader;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.io.OutputStream;

public class SshExample() {
// A buffered reader so we can request information from the user
private static BufferedReader reader =
new BufferedReader(new InputStreamReader(System.in));

public static void main(String args[]) {
try {
// Further code will be added here

} catch(Exception e) {
e.printStackTrace();
}
}
}

The next few sections will guide you through making the initial connection, authenticating the user and executing
a command or starting the users shell for a simple console based SSH application.

Making the initial connection

To create an SshClient instance import the class into your implementation class file and use the following code
to connect to an SSH server on the standard port 22.

SshClient ssh
= new SshClient();

System.out.print("Host to connect: ");

String hostname = reader.readLine();

ssh.connect(hostname);

When the client connects to the server, the server supplies its public key for the client to verify. You will see
that calling the connect method prompts the user within the console to verify the key:

The host firestar is currently unknown to the system
The host key fingerprint is: 1028: 69 54 9c 49 e5 92 59 40 5 66 c5 2e 9d 86 af ed
Do you want to allow this host key? [Yes|No|Always]:

In the default implementation of the connect method, J2SSH reads the $HOME/.ssh/known_hosts file to determines to which hosts connections
may be allowed. This is provided by the class ConsoleKnownHostsKeyVerification and the
default behavior can be emulated by the following code:

import com.sshtools.j2ssh.transport.IgnoreHostKeyVerification;

ssh.connect("firestar", new IgnoreHostKeyVerification());

When the connect method returns, the protocol has been negotiated and key exchange has taken place, leaving the connection ready for authenticating the user.

Authenticating the user

Once the connection has been completed the user is required to provide a set of credentials for authentication.
All client side authentication methods are implemented using the abstract class:

import com.sshtools.j2ssh.authentication.SshAuthenticationClient.

To perform authentication, the SshClient class provides the following method:

public int authenticate(SshAuthenticationClient auth);

There are currently five authentication methods implemented by J2SSH, 'password', 'publickey', 'keyboard-interactive' and 'hostbased'. With an extra agent authentication method that performs public key authentication using the J2SSH key agent.

Password Authentication

Password authentication is ideal for first time users as it requires no additional configuration within the SSH
client or server. The user simply supplies his username and password to the client which is then transmitted over
the encrypted connection to the server. The server then checks that the given password is acceptable to the native
password-authentication mechanism of the host operating system and returns the result to the client.

J2SSH implements the 'password' authentication method with the following class:

import com.sshtools.j2ssh.authentication.PasswordAuthenticationClient

Using the password authentication method is straight forward; create an instance of the PasswordAuthentication
class, set the username and password and pass to the SshClient to complete the authentication.

/**
* Create a PasswordAuthenticationClient instance, set the properties
* and pass to the SessionClient to authenticate
*/
PasswordAuthenticationClient pwd = new PasswordAuthenticationClient();

System.out.print("Username: ");
String username = reader.readLine();
auth.setUsername(username);

System.out.print("Password: ");
String password = reader.readLine();
auth.setPassword(password);

int result = ssh.authenticate(pwd);

The Authentication Result

When the authentication method completes it returns the result of the authentication. This integer value can be any
of the following three values defined in the class:

import com.sshtools.j2ssh.authentication.AuthenticationProtocolState;
..
..
if(result==AuthenticationProtocolState.FAILED)
System.out.println("The authentication failed");

if(result==AuthenticationProtocolState.PARTIAL)
System.out.println("The authentication succeeded but another"
+ "authentication is required");

if(result==AuthenticationProtocolState.COMPLETE)
System.out.println("The authentication is complete");

Retrieving the available authentication Methods

It is possible at any time after the connection has been established and before authentication has been completed to request a list of authentication methods that can be used. The getAvailableAuthMethods method returns a list of authentication method names.

public List getAvailableAuthMethods(String username);

It should be noted that the SSH specification allows the server to return authentication methods that are not valid
for the user.

Prompting the User for Authentication Details?

Each SshAuthenticationClient implementation can optionally be set a prompt interface which allows the user to
be prompted for the information once the authenticate method has been invoked.

public interface SshAuthenticationPrompt {
public boolean showPrompt(SshAuthenticationClient instance)
throws AuthenticationProtocolException;
}
}

The showPrompt method is called if the authentication instance is not ready to authenticate. The methods is called and the developer should verify the instance of the SshAuthenticationClient to make sure that it is compatible with the prompt (for example you cannot perform public key authentication
with a password prompt!). The user should then be duly prompted for the information and the instance set with the user's information. Once
complete, the prompt returns true to indicate that the user successfully entered correct information.

There are several prompts provided in the J2SSH common packages that provide useful Swing based dialogs to
prompt the user.

import com.sshtools.common.authentication.PasswordAuthenticationDialog;

/***
* Create a PasswordAuthenticationDialog instance and call the
* showAuthenticationMethod so the user can graphically
* enter their username and password
*/

PasswordAuthenticationClient pwd =
new PasswordAuthenticationClient();

PasswordAuthenticationDialog dialog =
new PasswordAuthenticationDialog(parent);

pwd.setAuthenticationPrompt(dialog);

int result;
result = ssh.authenticate(pwd);

Using a session channel to execute a command or start the users shell

Once the user is authenticated you will probably want to do something such as execute a command or start the users shell. The SSH protocol provides multiplexed channels over a single connection and the session channel is one of the channels defined by the SSH protocol. The session channel allows the client to execute a single command on the remote host and to communicate with the process by sending and receiving data. The J2SSH SessionChannelClient implements this channel and we will use this to execute a basic "ls" command.

The session channel provides an inputstream and outputstream for reading/writing, but before we can do this we need to setup the channel for our command. First we open the channel itself by calling the SshClient method:

SessionChannelClient session = ssh.openSessionChannel();

Now that we have a session instance we need to configure it for our command, there are several options that can be set before we invoke one of the methods that will start the session.

Setting Environment VariablesThe SSH protocol provides a method to set an environment variable for the processes environment, the protocol also leaves the actual implementation of this down the server implementation, and in our experience most servers do not allow for this for security reasons. However the method is available since its defined in the protocol specification, so your free to try to use it, but beware the variable may not be set!

public boolean setEnvironmentVariable(String name, String value);

Requesting a Pseudo Terminal
A pseudo terminal is a device that imitates a terminal. Rather than being connected to an actual terminal, a pseudo-terminal (or pty) is connected to a process. If the command you are executing is expecting a terminal (such as a shell command) you can request that a pseudo terminal be attached to the process by calling the requestPseudoTerminal method.

public boolean requestPseudoTerminal(String term, int cols, int rows, int width, int height, String modes);

Invoking a commandAfter the above operations have been performed you can then request that the session either start the user's shell, execute a specific command or start an SSH subsystem (such as SFTP). You should not invoke a subsystem unless you are able to read/write the subsystem protocol, there are many additional utilities within J2SSH that provide for the available subsystems.

To start the users default shell use:

public boolean startShell();

Or to execute a specific command use:

public boolean executeCommand(String command);

An important note to remember is that this does not execute a shell command. You cannot for instance issue the command executeCommand("dir")" on the Windows Operating system as this is a shell command, instead use "cmd.exe /C dir". This method executes a binary executable and so should be used to execute any program other than the users shell.

Handling Session Data
Once the session has been configured and a command or shell has been started, you can begin to
transfer data to and from the remote computer using the sessions IO streams. These streams provide you with a
standardized interface for reading and writing the data.

The Session Channel's OutputStream
The format of writing data varies according to how you configured the session, for example if you executed the
users shell then the data should be written as if the user had entered the commands interactively.

/** Writing to the session OutputStream */

OutputStream out = session.getOutputStream();

String cmd = "ls\n";

out.write(cmd.getBytes());

The Session Channel's InputStream

/**
* Reading from the session InputStream
*/
InputStream in = session.getInputStream();
byte buffer[] = new byte[255];
int read;
while((read = in.read(buffer)) > 0) {
String out = new String(buffer, 0, read);
System.out.println(out);
}

Reading from stderr
The session also provides the stderr data provided by the remote session. Again an InputStream is provided.

public InputStream session.getStderrInputStream();

Closing the Session

The session can be closed using the following method:

public void close();

Disconnecting

The connection can be terminated by either side. To terminate the connection call the SshClient method:

public void disconnect();

A word on executing multiple commands

So we can now execute a single command on the remote server, but what's that I hear you say? I want to execute more than one command? Well if you cast your mind back I told you that the SSH protocol provides multiplexed channels over a single connection, so executing another command is as simple as executing the first, just create a new instance of the SessionChannelClient for every command you want to execute. You can execute them simultaneously or one after another, but always create a new session (since the session is closed when the command finishes and the protocol does not allow for re-using of a session to execute another command).

There is a drawback to this in that the process environment is not passed on from one session to another, so you cannot for example execute a command to change directory and then another to execute a script in that directory, since the change directory is lost when the session closes and the new command starts back in the default working directory. Of course you could always put the cd command into the script? Or use the shell to execute both commands. This subject is certainly a bit more advanced so I will leave it for another day and pencil in a new article to discuss all the alternatives for executing multiple commands.

This concludes our getting started tutorial, you should now have a basic working knowledge of how to connect, authenticate and execute commands using J2SSH.

Nice reply

2012-04-08T22:48:00.001-07:00

A reply from CEO of J.P. Morgan to a pretty girl seeking a rich husband

A young and pretty lady posted this on a popular forum:

Title: What should I do to marry a rich guy?

I'm going to be honest of what I'm going to say here.

I'm 25 this year. I'm very pretty, have style and good taste. I wish to marry a guy with $500k annual salary or above.

You might say that I'm greedy, but an annual salary of $1M is considered only as middle class in New York.

My requirement is not high. Is there anyone in this forum who has an income of $500k annual salary? Are you all married?

I wanted to ask: what should I do to marry rich persons like you?

Among those I've dated, the richest is $250k annual income, and it seems that this is my upper limit.

If someone is going to move into high cost residential area on the west of New York City Garden(?), $250k annual income is not enough.

I'm here humbly to ask a few questions:
1) Where do most rich bachelors hang out? (Please list down the names and addresses of bars, restaurant, gym)
2) Which age group should I target?
3) Why most wives of the riches are only average-looking? I've met a few girls who don't have looks and are not interesting, but they are able to marry rich guys.

4) How do you decide who can be your wife, and who can only be your girlfriend? (my target now is to get married)

Ms. Pretty

A philosophical reply from CEO of J.P. Morgan:

Dear Ms. Pretty,
I have read your post with great interest. Guess there are lots of girls out there who have similar questions like yours. Please allow me to analyse your situation as a professional investor.

My annual income is more than $500k, which meets your requirement, so I hope everyone believes that I'm not wasting time here.

From the standpoint of a business person, it is a bad decision to marry you. The answer is very simple, so let me explain.

Put the details aside, what you're trying to do is an exchange of "beauty" and "money" : Person A provides beauty, and Person B pays for it, fair and square.

However, there's a deadly problem here, your beauty will fade, but my money will not be gone without any good reason. The fact is, my income might increase from year to year, but you can't be prettier year after year.

Hence from the viewpoint of economics, I am an appreciation asset, and you are a depreciation asset. It's not just normal depreciation, but exponential depreciation. If that is your only asset, your value will be much worse 10 years later.

By the terms we use in Wall Street, every trading has a position, dating with you is also a "trading position".
If the trade value dropped we will sell it and it is not a good idea to keep it for long term - same goes with the marriage that you wanted. It might be cruel to say this, but in order to make a wiser decision any assets with great depreciation value will be sold or "leased".

Anyone with over $500k annual income is not a fool; we would only date you, but will not marry you. I would advice that you forget looking for any clues to marry a rich guy. And by the way, you could make yourself to become a rich person with $500k annual income.This has better chance than finding a rich fool.

Hope this reply helps.

Married or not you should read this...

2012-04-04T23:39:00.003-07:00

“When I got home that night as my wife served dinner, I held her hand and said, I’ve got something to tell you. She sat down and ate quietly. Again I observed the hurt in her eyes.

Suddenly I didn’t know how to open my mouth. But I had to let her know what I was thinking. I want a divorce. I raised the topic calmly. She didn’t seem to be annoyed by my words, instead she asked me softly, why?

I avoided her question. This made her angry. She threw away the chopsticks and shouted at me, you are not a man! That night, we didn’t talk to each other. She was weeping. I knew she wanted to find out what had happened to our marriage. But I could hardly give her a satisfactory answer; she had lost my heart to Jane. I didn’t love her anymore. I just pitied her!

With a deep sense of guilt, I drafted a divorce agreement which stated that she could own our house, our car, and 30% stake of my company. She glanced at it and then tore it into pieces. The woman who had spent ten years of her life with me had become a stranger. I felt sorry for her wasted time, resources and energy but I could not take back what I had said for I loved Jane so dearly. Finally she cried loudly in front of me, which was what I had expected to see. To me her cry was actually a kind of release. The idea of divorce which had obsessed me for several weeks seemed to be firmer and clearer now.

The next day, I came back home very late and found her writing something at the table. I didn’t have supper but went straight to sleep and fell asleep very fast because I was tired after an eventful day with Jane. When I woke up, she was still there at the table writing. I just did not care so I turned over and was asleep again.

In the morning she presented her divorce conditions: she didn’t want anything from me, but needed a month’s notice before the divorce. She requested that in that one month we both struggle to live as normal a life as possible. Her reasons were simple: our son had his exams in a month’s time and she didn’t want to disrupt him with our broken marriage.

This was agreeable to me. But she had something more, she asked me to recall how I had carried her into out bridal room on our wedding day. She requested that every day for the month’s duration I carry her out of our bedroom to the front door ever morning. I thought she was going crazy. Just to make our last days together bearable I accepted her odd request.

I told Jane about my wife’s divorce conditions. . She laughed loudly and thought it was absurd. No matter what tricks she applies, she has to face the divorce, she said scornfully.

My wife and I hadn’t had any body contact since my divorce intention was explicitly expressed. So when I carried her out on the first day, we both appeared clumsy. Our son clapped behind us, daddy is holding mommy in his arms. His words brought me a sense of pain. From the bedroom to the sitting room, then to the door, I walked over ten meters with her in my arms. She closed her eyes and said softly; don’t tell our son about the divorce. I nodded, feeling somewhat upset. I put her down outside the door. She went to wait for the bus to work. I drove alone to the office.

On the second day, both of us acted much more easily. She leaned on my chest. I could smell the fragrance of her blouse. I realized that I hadn’t looked at this woman carefully for a long time. I realized she was not young any more. There were fine wrinkles on her face, her hair was graying! Our marriage had taken its toll on her. For a minute I wondered what I had done to her.

On the fourth day, when I lifted her up, I felt a sense of intimacy returning. This was the woman who had given ten years of her life to me. On the fifth and sixth day, I realized that our sense of intimacy was growing again. I didn’t tell Jane about this. It became easier to carry her as the month slipped by. Perhaps the everyday workout made me stronger.

She was choosing what to wear one morning. She tried on quite a few dresses but could not find a suitable one. Then she sighed, all my dresses have grown bigger. I suddenly realized that she had grown so thin, that was the reason why I could carry her more easily.

Suddenly it hit me… she had buried so much pain and bitterness in her heart. Subconsciously I reached out and touched her head.

Our son came in at the moment and said, Dad, it’s time to carry mom out. To him, seeing his father carrying his mother out had become an essential part of his life. My wife gestured to our son to come closer and hugged him tightly. I turned my face away because I was afraid I might change my mind at this last minute. I then held her in my arms, walking from the bedroom, through the sitting room, to the hallway. Her hand surrounded my neck softly and naturally. I held her body tightly; it was just like our wedding day.

But her much lighter weight made me sad. On the last day, when I held her in my arms I could hardly move a step. Our son had gone to school. I held her tightly and said, I hadn’t noticed that our life lacked intimacy. I drove to office…. jumped out of the car swiftly without locking the door. I was afraid any delay would make me change my mind…I walked upstairs. Jane opened the door and I said to her, Sorry, Jane, I do not want the divorce anymore.

She looked at me, astonished, and then touched my forehead. Do you have a fever? She said. I moved her hand off my head. Sorry, Jane, I said, I won’t divorce. My marriage life was boring probably because she and I didn’t value the details of our lives, not because we didn’t love each other anymore. Now I realize that since I carried her into my home on our wedding day I am supposed to hold her until death do us apart. Jane seemed to suddenly wake up. She gave me a loud slap and then slammed the door and burst into tears. I walked downstairs and drove away. At the floral shop on the way, I ordered a bouquet of flowers for my wife. The salesgirl asked me what to write on the card. I smiled and wrote, I’ll carry you out every morning until death do us apart.

That evening I arrived home, flowers in my hands, a smile on my face, I run up stairs, only to find my wife in the bed -dead. My wife had been fighting CANCER for months and I was so busy with Jane to even notice. She knew that she would die soon and she wanted to save me from the whatever negative reaction from our son, in case we push through with the divorce.— At least, in the eyes of our son—- I’m a loving husband….

The small details of your lives are what really matter in a relationship. It is not the mansion, the car, property, the money in the bank. These create an environment conducive for happiness but cannot give happiness in themselves.

So find time to be your spouse’s friend and do those little things for each other that build intimacy. Do have a real happy marriage!

If you don’t share this, nothing will happen to you.

If you do, you just might save a marriage. Many of life’s failures are people who did not realize how close they were to success when they gave up

The Elephant Rope

2012-04-04T23:19:00.002-07:00

As a man was passing the elephants, he suddenly stopped, confused by the fact that these huge creatures were being held by only a small rope tied to their front leg. No chains, no cages. It was obvious that the elephants could, at anytime, break away from their bonds but for some reason, they did not.

He saw a trainer nearby and asked why these animals just stood there and made no attempt to get away. “Well,” trainer said, “when they are very young and much smaller we use the same size rope to tie them and, at that age, it’s enough to hold them. As they grow up, they are conditioned to believe they cannot break away. They believe the rope can still hold them, so they never try to break free.”

The man was amazed. These animals could at any time break free from their bonds but because they believed they couldn’t, they were stuck right where they were.

Like the elephants, how many of us go through life hanging onto a belief that we cannot do something, simply because we failed at it once before?

Failure is part of learning; we should never give up the struggle in life.

Java Threads

2012-04-03T23:38:00.001-07:00

This post is the first of set of posts about Java Concurrency.

The concurrency is the fact to made several things at the same time using several threads.

A thread, also called Lightweight Process, is treatment unity. Threads executes code in parralel of each other threads currently running. When you’ve only one processor, there is a thread running at the same time of the others, you only have the impression of concurrency (I don’t say it’s not useful, I say it’s different), but when you’ve multiple processors, you’ll see the power of multi threading. In this case, you can have your threads distributed on the processors of the computer.

In Java, a thread is an instance of the class java.lang.Thread. A Thread can be managed in one of these two ways :

Directly mapped to a native thread of the operating system. This is used when the operating system provide a preemptive threading system.
Managed by the virtual machine in a premptive way.

A preemptive system, is a system in which the threads are managed by a scheduler and can be interrupted at any time to give processor to an other thread. When you program, you doesn’t have to pay attention of which type of threads you use, the result will normally be the same. But you’ve to know that there can differences between operating systems.

There is three very important concepts when doing concurrent programming :

Atomicity : An operation is said atomic when it cannot be interrupted. There is almost no atomic operations in Java, the only we’ve is the assignation a = 5, but a = b++ is not atomic. In some cases, you’ll have to make atomic some actions with synchronization, we’ll see later how to do that.
Visibility : This occurs when a thread must watch the actions of an other threads by example the termination of the thread. This also implies some kind of synchronization.
Order of execution : When you have normal program, all you lines of code run in the same order every time you launch the application. This is not the case when you make concurrent programming. You first instruction can followed by an instruction of the thread B ou by the first instruction. And that can change every time you launch the application. The order of execution is not guaranteed ! I will certainly repeat that sometimes, but that’s important to know.

We’ll see these concepts more deeply in the others parts of the set.

Lets start introducing the Thread class in Java. You can create threads in two ways :

Extends Thread
Implements Runnable and pass an instance of your news class to the Thread constructor

The first solution isn’t a good solution because what you’re creating is not a new specialized thread, but severals instructions to run in a new Thread, namely a Runnable. Implementing Runnable is also better because Runnable is an interface and so, you can also extends a class and implementing Runnable, that’s useful in some cases.

In my examples, I’ll always use the first way. So let’s declare our first Runnable :

public class MyFirstRunnable implements Runnable{
    @Override
    public void run() {
        System.out.println("In a thread");
    }
}

And use it to create a new Thread and start it :

Thread thread = new Thread(new MyFirstRunnable());
thread.start();

The Thread will stopped when the end of the run() will be reached. You cannot force a thread to stop (there is stop() method, but deprecated), we’ll see later how to properly stop a thread.

And now, what happens if we add a simple line of code to our program :

Thread thread = new Thread(new MyFirstRunnable());
thread.start();
System.out.println("In the main Thread");

Can you predict the result of this code ? Nobody can’t, it’s not predictable, you can have :

In a thread
In the main Thread

In the main Thread
In a thread

And we cannot do better than that.

You can use the Runnable several times :

Runnable runnable = new MyFirstRunnable();
 
for(int i = 0; i &lt; 25; i++){
    new Thread(runnable).start();
}

Now, 25 threads are launched.

You can also give names to Thread using the setName() method. You can get the name of the current thread using Thread.currentThread().getName(). Let’s do a little example :

public class MySecondRunnable implements Runnable{
    @Override
    public void run() {
        System.out.printf("I'm running in thread %s \n", Thread.currentThread().getName());
    }
}

Runnable runnable = new MySecondRunnable();
 
for(int i = 0; i &lt; 25; i++){
    Thread thread = new Thread(runnable);
    thread.setName("Thread " + i);
    thread.start();
}

This is the best example to see that the other is unpredictable. Here are two executions on my machine :

I'm running in thread Thread 0
I'm running in thread Thread 1
I'm running in thread Thread 2
I'm running in thread Thread 3
I'm running in thread Thread 4
I'm running in thread Thread 5
I'm running in thread Thread 7
I'm running in thread Thread 14
I'm running in thread Thread 13
I'm running in thread Thread 12
I'm running in thread Thread 11
I'm running in thread Thread 10
I'm running in thread Thread 9
I'm running in thread Thread 8
I'm running in thread Thread 6
I'm running in thread Thread 15
I'm running in thread Thread 16
I'm running in thread Thread 17
I'm running in thread Thread 18
I'm running in thread Thread 19
I'm running in thread Thread 20
I'm running in thread Thread 21
I'm running in thread Thread 22
I'm running in thread Thread 23
I'm running in thread Thread 24

I'm running in thread Thread 0
I'm running in thread Thread 1
I'm running in thread Thread 2
I'm running in thread Thread 3
I'm running in thread Thread 4
I'm running in thread Thread 5
I'm running in thread Thread 6
I'm running in thread Thread 7
I'm running in thread Thread 8
I'm running in thread Thread 9
I'm running in thread Thread 10
I'm running in thread Thread 11
I'm running in thread Thread 12
I'm running in thread Thread 13
I'm running in thread Thread 14
I'm running in thread Thread 15
I'm running in thread Thread 16
I'm running in thread Thread 17
I'm running in thread Thread 18
I'm running in thread Thread 19
I'm running in thread Thread 20
I'm running in thread Thread 21
I'm running in thread Thread 22
I'm running in thread Thread 23
I'm running in thread Thread 24

Like you can see, the order the threads instructions are executed is not guaranteed at all.

So here we are with the first part of this suite of articles about Java Concurrency. In the next post, we’ll see the operations you can make directly on threads (stopping, joining, sleeping, …).

I hope you found that post interesting.

we’ll see in this article what we can do to manipulate Threads.

When we’ve Threads, we can make several operations on the Threads :

Make the current Thread sleeping during x milliseconds
Wait for an other thread to complete
Manage the priorities of Threads and pause a thread to give an other thread the opportunity to run
Interrupt a thread

We’ll now see how to do all these things.

First and easier, we can make a thread sleeping for a certain number of milliseconds. To do that, the Thread class has a method sleep(long millis). But this method is static, so you can only make the current Thread sleeping. You cannot choose the thread you want to make sleeping, your only choice is the current Thread so :

Thread.sleep(1000);

makes the current Thread sleep during 1000 milliseconds (1 second). But, you have to catch an exception, InterruptedException. This exception occurs if the sleeping thread is interrupted. So you have to do that :

try {
 Thread.sleep(1000);
} catch (InterruptedException e){
 e.printStackTrace();
}

But this not the good way to manage the InterruptedException. We’ll see in one moment, how to deal with this exception.

If you want more precision, you can use the overloaded version of sleep that takes the number of millis plus a certain number of nanos to sleep. The precision of this sleep depends on the system timers and clocks.

For example, if you want to sleep 1000 milliseconds and 1000 nanoseconds (1 microsecond), you can do like that :

try {
 Thread.sleep(1000, 1000);
} catch (InterruptedException e){
 e.printStackTrace();
}

Here is a little example to test that :

public class SleepThread {
    public static void main(String[] args) {
        System.out.println("Current time millis : " + System.currentTimeMillis());
 
        try {
            Thread.sleep(1000);
        } catch (InterruptedException e) {
            e.printStackTrace();
        }
 
        System.out.println("Current time millis : " + System.currentTimeMillis());
 
        System.out.println("Nano time : " + System.nanoTime());
 
        try {
            Thread.sleep(2, 5000);
        } catch (InterruptedException e) {
            e.printStackTrace();
        }
 
        System.out.println("Nano time : " + System.nanoTime());
    }
}

In my computer, this produce this result :

Current time millis : 1273959308480
Current time millis : 1273959309480
Nano time : 5878165216075
Nano time : 5878166730976

You can see that the sleep of millis is very precise, but with nanos the result can vary a lot. And of course, the result depends of your computer, your operating system and your configuration.

An other thing, you can do with Threads, is waiting for an other thread to die. For example, you can create five thread to compute sub result and wait for these 5 threads to finish to compute the final results based on the results of the five threads. To do that, you can use the join() method of the Thread class. This method is not static, so you can use it on any thread to wait for it to die. Like sleep() this method throws InterruptedException in the when the thread is interrupted during waiting for an other thread. So to wait on thread2, you just have to do that :

try {
 thread2.join();
} catch (InterruptedException e){
 e.printStackTrace();
}

That will make the current Thread waiting for thread2 to die. You can also add a timeout in millis, or millis + nanos, with the overloaded versions of join(), join(long millis) and join(long millis, int nanos). Here is little example that shows all that stuff :

public class JoinThread {
 public static void main(String[] args) {
  Thread thread2 = new Thread(new WaitRunnable());
  Thread thread3 = new Thread(new WaitRunnable());
 
  System.out.println("Current time millis : " + System.currentTimeMillis());
 
  thread2.start();
 
  try {
   thread2.join();
  } catch (InterruptedException e) {
   e.printStackTrace();
  }
 
  System.out.println("Current time millis : " + System.currentTimeMillis());
 
  thread3.start();
 
  try {
   thread3.join(1000);
  } catch (InterruptedException e) {
   e.printStackTrace();
  }
 
  System.out.println("Current time millis : " + System.currentTimeMillis());
 }
 
 private static class WaitRunnable implements Runnable {
  @Override
  public void run() {
   try {
    Thread.sleep(5000);
   } catch (InterruptedException e) {
    e.printStackTrace();
   }
  }
 }
}

That produce this result on my computer :

Current time millis : 1274015478535
Current time millis : 1274015483538
Current time millis : 1274015484538

You can see that the first join() wait 5 seconds for the other thread and when we set a timeout, we wait only 1 seconds and return from join method.

When working with Threads, it’s also possible to change the priority of a Thread. In the Java Virtual Machine, the Thread scheduler, use a priority-based scheduling. So if a Thread enter in Runnable state with a higher priority than the running Thread, the new Thread will run and the current running thread will return to Runnable state and waits for its turn. But this behaviour is not guaranteed and is completely depending on the virtual machine you are working on. So, do not rely on thread priorities, just use them to improve efficiency of your program.

Normally, the priority range of Threads is an integer from 0 to 10, but some virtual machine have lower or higher ranges. To know the range of priority, you can use constants of the Thread class :

public class ThreadPriorityRange {
 public static void main(String[] args) {
  System.out.println("Minimal priority : " + Thread.MIN_PRIORITY);
  System.out.println("Maximal priority : " + Thread.MAX_PRIORITY);
  System.out.println("Norm priority : " + Thread.NORM_PRIORITY);
 }
}

On my computer, I’ve the most current values :

Minimal priority : 1
Maximal priority : 10
Norm priority : 5

To set the priority of a Thread, you can use the setPriority(int priority) method of the Thread class. If you enter a value greater than the maximal priority, the maximal value will be used. If you don’t specify a priority, the used priority, will be the priority of the current Thread.

An other way to works with priority is the yield() method. This method is static, so this works on the current Thread. The purpose of this method is to make the Thread going to Runnable again and to give the opportunity to other threads to get their turn. But in practice, the behaviour of this method is not guaranteed. It can be implemented as a no-op on certain systems. It’s not easy to test that in practice, because the results can truly depends on your computer, virtual machine and operating system. It’s a good things to not use the priorities of Threads in practice.

The last thing you can do with a Thread, is to interrupt it. In Java, you have no way to force a Thread to stop, if the Thread is not well-done, it can continue its execution infinitely. But you can interrupt a Thread with the interrupt() method. This method interrupt the thread, if the thread is sleeping or joining an other Thread, an InterruptedException is thrown. You have to know that if the thread was sleeping or joining, the interrupted status of the Thread will be cleared. Namely, the method isInterrupted() will return false. A little example to demonstrate that :

public class InterruptThread {
 public static void main(String[] args) {
  Thread thread1 = new Thread(new WaitRunnable());
 
  thread1.start();
 
  try {
   Thread.sleep(1000);
  } catch (InterruptedException e) {
   e.printStackTrace();
  }
 
  thread1.interrupt();
 }
 
 private static class WaitRunnable implements Runnable {
  @Override
  public void run() {
   System.out.println("Current time millis : " + System.currentTimeMillis());
 
   try {
    Thread.sleep(5000);
   } catch (InterruptedException e) {
    System.out.println("The thread has been interrupted");
    System.out.println("The thread is interrupted : " + Thread.currentThread().isInterrupted());
   }
 
   System.out.println("Current time millis : " + System.currentTimeMillis());
  }
 }
}

That produce that kind of result :

Current time millis : 1274017633151
The thread has been interrupted
The thread is interrupted : false
Current time millis : 1274017634151

You can see that after one second, the second thread is interrupted and that the interrupted status has been set to false. If you are not sleeping, but making a lot of heavy actions, you can test for interrupt like that to make your thread correctly interrupts :

public class InterruptableRunnable implements Runnable {
 @Override
 public void run() {
  while(!Thread.currentThread().isInterrupted()){
   //Heavy operation
  }
 }
}

Now that you know how to interrupt a thread, you can imagine, that simply catch the InterruptedException is not enough to make your thread “interrupt safe”. Imagine that your thread something like that :

public class UglyRunnable implements Runnable {
 @Override
 public void run() {
  while(!Thread.currentThread().isInterrupted()){
   //Heavy operation
   try {
    Thread.sleep(5000);
   } catch (InterruptedException e) {
    e.printStackTrace();
   }
   //Other operation
  }
 }
}

And now, an other thread want to interrupt your thread while your thread is sleeping. The sleep will be interrupted, but the interrupted status will be cleared so the loop will continue. A solution to make a better thread is to interrupt again the thread after an InterruptException :

public class BetterRunnable implements Runnable {
 @Override
 public void run() {
  while(!Thread.currentThread().isInterrupted()){
   //Heavy operation
   try {
    Thread.sleep(5000);
   } catch (InterruptedException e) {
    Thread.currentThread().interrupt();
   }
   //Other operation
  }
 }
}

With that code, the interrupted status will be restored and the loop will be stopped after interrupt. Depending on your code, you can also add a continue statement after the interrupt() to not make operations after interrupt. In some cases, you’ll also needs to make several if statements to test the interrupted status to do or not to do some operations.

So, we’ve now covered the main operations you can do on threads. I hope you found this article interesting.

After learning how to create threads and manipulate them, it’s time to go to most important things : synchronization.

Synchronization is a way to make some code thread safe. A code that can be accessed by multiple threads must be made thread safe. Thread Safe describe some code that can be called from multiple threads without corrupting the state of the object or simply doing the thing the code must do in right order.

For example, we can take this little class :

public class Example {
    private int value = 0;    
 
    public int getNextValue(){
        return value++;
    }
}

It’s really simple and works well with one thread, but absolutely not with multiple threads. An incrementation like this is not a simple action, but three actions :

Read the current value of “value”
Add one to the current value
Write that new value to “value”

Normally, if you have two threads invoking the getNextValue(), you can think that the first will get 1 and the next will get 2, but it is possible that the two threads get the value 1. Imagine this situation :

Thread 1 : read the value, get 0, add 1, so value = 1
Thread 2 : read the value, get 0, add 1, so value = 1
Thread 1 : write 1 to the field value and return 1
Thread 2 : write 1 to the field value and return 1

These situations come from what we call interleaving. Interleaving describe the possible situations of several threads executing some statements. Only for three operations and two threads, there is a lot of possible interleavings.

So we must made the operations atomic to works with multiple threads. In Java, the first way to make that is to use a lock. All Java objects contains an intrinsic locks, we’ll use that lock to make methods or statement atomic. When a thread has a lock, no other thread can acquire it and must wait for the first thread to release the lock. To acquire the lock, you have to use the synchronized keyword to automatically acquire and release a lock for a code. You can add the synchronized keyword to a method to acquire the lock before invoking the method and release it after the method execution. You can refactor the getNextValue() method using the synchronized keyword :

public class Example {
    private int value = 0;    
 
    public synchronized int getNextValue(){
        return value++;
    }
}

With that, you have the guarantee that only thread can execute the method at the same time. The used lock is the intrinsic lock of the instance. If the method is static, the used lock is the Class object of Example. If you have two methods with the synchronized keyword, only one method of the two will be executed at the same time because the same lock is used for the two methods. You can also write it using a synchronized block :

public class Example {
    private int value = 0;
 
    public int getNextValue() {
        synchronized (this) {
            return value++;
        }
    }
}

This is exactly the same as using the synchronized keyword on the method signature. Using synchronized blocks, you can choose the lock to block on. By example, if you don’t want to use the intrinsic lock of the current object but an other object, you can use an other object just as a lock :

public class Example {
    private int value = 0;
 
    private final Object lock = new Object();
 
    public int getNextValue() {
        synchronized (lock) {
            return value++;
        }
    }
}

The result is the same but has one difference, the lock is internal to the object so no other code can use the lock. With complex classes, it not rare to use several locks to provide thread safety on the class.

There is an other issue with multiple threads : the visibility of the variables. This seems when a change made by a thread is visible by an other thread. For performance improvements, the Java compiler and virtual machines can made some improvements using registers and cache. By default, you have no guarantee that a change made by a thread is visible to an other thread. To make a change visible to an other thread, you must use synchronized blocks to ensure visibility of the change. You must use synchronized blocks for the read and for the write of the shared values. You must make that for every read/write of a value shared between multiple threads.

You can also use the volatile keyword on the field to ensure the visibility of read/write between multiple threads. The volatile keyword ensure only visibility, not atomicity. The synchronized blocks ensure visibility and atomicity. So you can use the volatile keyword on fields that doesn’t need atomicity (if you make only read and write to the field without depending on the current value of the field by example).

You can also note that this simple example can be solved using AtomicInteger, but that will be covered later in an other part of the posts.

Pay attention that trying to solve thread safety on a problem can add new issues of deadlock. By example, if thread A owns the lock 1 and are waiting for the lock 2 and if lock 2 is acquired by thread B who waits on lock 1, there is a deadlock. Your program is dead. So you have to pay great attention to the locks.

There is several rules that we must keep in mind when using locks :

Every mutable fields shared between multiple threads must be guarded with a lock or made volatile, if you only need visibility
Synchronize only the operations that must synchronized, this improve the performances. But don’t synchronize too few operations. Try to keep the lock only for short operations.
Always know which locks are acquired and when there are acquired and by which thread
An immutable object is always thread safe

We continue with the Java Concurrency thema with the semaphores. Semaphores is also a way to synchronize threads.

Semaphores are a really simple concept, invented by the famous Dutch computer scientist Edsger Dijkstra. Basically a semaphore is a counter (integer) that allows a thread to get into a critical region if the value of the counter is greater than 0. If it’s the case, the counter is decremented by one otherwise, the thread is waiting until it can go. And when the thread go away from the critical region, the counter is incremented by one to allow one more thread to pass the critical region. A semaphore is created with a certain value for its counter. So, you can execute two actions on a semaphore P and V.

By example, if you have a critical that cannot be executed concurrently, you can use a semaphore :

sem mutex = new sem(1)
 
P(mutex)
//Critical region
V(mutex)

So you must always call by yourself the P operation before the critical region and V after it. We call a mutex (mutual exclusion) a semaphore with a value of one. So only one thread can enter the region guarded by the semaphore. This is the most used semaphore. The other use of semaphore is to guard a set of resources like database connections or a data pool.

In Java, a semaphore is created using the java.util.concurrent.Semaphore class. You can create easily :

Semaphore mutex = new Semaphore(1);
Semaphore available = new Semaphore(100);

The P and V operations are represented using the acquire and release methods. The method acquire can be interrupted if the thread is interrupted. There is an ininterruptible version with the method acquireUninterruptibly(). There is also a third version with the tryAcquire method. This method acquire a permit only if there is one permit available, otherwise, this method return false directly. All the waiting methods have also an overloaded version with a timeout. You can also acquire several permits at once using the permits argument to the different versions of acquire methods.

A little example with a mutex using the same example as the previous post on Java concurrency :

public class Example {
    private int value = 0;
 
    private final Semaphore mutex = new Semaphore(1)
 
    public int getNextValue() throws InterruptedException {
        try {
            mutex.acquire();
            return value++;
        } finally {
            mutex.release();
        }
    }
}

For more informations about Semaphore in Java, the best is to consult the Javadoc of the Semaphore class.

To conclude, semaphores are a powerful ways to solve concurrency problems, but this is not adapted to all problems. If you need only mutual exclusion, synchronized blocks are a better solutions. The problems with semaphores is that you can forget to call the release method and that can cause deadlock sometimes difficult to find.

Love Story 1

2012-04-03T23:32:00.002-07:00

A boy proposed his girlfriend for marriage,

Girl: Tell me… who do you love most in this world..?

Boy: You, of course!
Girl: what am I to you?

Boy: The boy thought for a moment and looked intently in her eyes and said
“u r missing part of my heart”

She smiled,and she accepted his proposal.

After their wedding, the couple had a sweet and happy life for a while.However,t he youthful couple began to drift apart due to the busy schedule of life and the never-ending worries of daily problems…their life became difficult.
:

All the challenges posed by the harsh realities of life began to drift away deir dreams and love for each other…The couple began to have more quarrels and each quarrel became make their relation more worst.

One day, after the quarrel, the girl ran out of the house…. At the opposite side of the road, she shouted,”You don’t love me…!”

The boy hated her childishness and out of impulse, said,”May be, it was a mistake for us to be together..!

You were never da missing part of my …!”

Suddenly, she turned quiet and stood there for a long while…. He regretted what he said but words spoken cant be taken back.

With tears in her eyes, she went home to pack her things and Before leaving the house,she left a note for him:
“If I’m really not da missing part of ur love, let me go..n search for sum1 whu is…It is less painful this way… let us go on our separate ways and search for our own partners…”
Five years went by….

He never remarried but he had tried to find out about her life indirectly….. . She had left the country and was living her dreams.

He use to regret on what he did but never tried to bring her back.. In the dark and lonely night, he lit his cigarette and felt the lingering ache in his heart.He couldn’t bring himself to admit that he was missing her…..

One day, they finally met…. At the airport, He was going away on a business trip .He saw her,She was standing there alone, with just the security door separating them.

She smiled at him gently.

Boy: How are you ?

Girl: I’m fine. How about you… Have you found your heart’s missing part…?

Boy: No.

Girl: I’ll be flying to New York in the next flight.

Boy: I’ll be back in 2 weeks time. Give me a call when you are back…You know my number…. Nothing has changed.

With a smile, she turned around and waved good-bye..
“Good-bye . . .”

Same evening he heard of a plane crash which was headed to New York.He tried to know about her and found that,She died.

Midnight….. Once again, he lit his cigarette….. And like before, he felt the lingering ache in his heart…. He finally knew, she was that missing part that he had carelessly broken . . .

Sometimes, people say things out of moments of fury…..

We take out our 99% frustrations at our loved ones…. And even though we know that we ought to”think twice and act wisely”, we actually don’t do it. Things happen each day, many of which are beyond our control.

Try not to hurt ur loved once,because a moment of anger could b a lifetime punishment.. ? — with

Java excecutor

2012-04-03T23:20:00.002-07:00

Let’s start with a new post in the Java concurrency series.

This time we’ll learn how to start cleanly new threads and to manage thread pools. In Java, if you have a Runnable like this :

Runnable runnable = new Runnable(){
   public void run(){
      System.out.println("Run");
   }
}

You can easily run it in a new thread :

new Thread(runnable).start();

This is very simple and clean, but what if you’ve several long running tasks that you want to load in parralel and then wait for the completion of all the tasks, it’s a little bit harder to code and if you want to get the return value of all the tasks it becomes really difficult to keep a good code. But like for almost any problems, Java has a solution for you, the Executors. This simple class allows you to create thread pools and thread factories.

A thread pool is represented by an instance of the class ExecutorService. With an ExecutorService, you can submit task that will be completed in the future. Here are the type of thread pools you can create with the Executors class :

Single Thread Executor : A thread pool with only one thread. So all the submitted task will be executed sequentially. Method : Executors.newSingleThreadExecutor()
Cached Thread Pool : A thread pool that create as many threads it needs to execute the task in parralel. The old available threads will be reused for the new tasks. If a thread is not used during 60 seconds, it will be terminated and removed from the pool. Method : Executors.newCachedThreadPool()
Fixed Thread Pool : A thread pool with a fixed number of threads. If a thread is not available for the task, the task is put in queue waiting for an other task to ends. Method : Executors.newFixedThreadPool()
Scheduled Thread Pool : A thread pool made to schedule future task. Method :Executors.newScheduledThreadPool()
Single Thread Scheduled Pool : A thread pool with only one thread to schedule future task. Method :Executors.newSingleThreadScheduledExecutor()

Once you have a thread pool, you can submit task to it using the different submit methods. You can submit a Runnable or a Callableto the thread pool. The method return a Future representing the future state of the task. If you submitted a Runnable, the Future object return null once the task finished.

By example, if you have this Callable :

private final class StringTask implements Callable<String> {
   public String call(){
      //Long operations
 
      return "Run";
   }
}

If you want to execute that task 10 times using 4 threads, you can use that code :

ExecutorService pool = Executors.newFixedThreadPool(4);
 
for(int i = 0; i < 10; i++){
   pool.submit(new StringTask());
}

But you must shutdown the thread pool in order to terminate all the threads of the pool :

pool.shutdown();

If you don’t do that, the JVM risk to not shutdown because there is still threads not terminated. You can also force the shutdown of the pool using shutdownNow, with that the currently running tasks will be interrupted and the tasks not started will not be started at all.

But with that example, you cannot get the result of the task. So let’s get the Future objects of the tasks :

ExecutorService pool = Executors.newFixedThreadPool(4);
 
List<Future<String>> futures = new ArrayList<Future<String>>(10);
 
for(int i = 0; i < 10; i++){
   futures.add(pool.submit(new StringTask()));
}
 
for(Future<String> future : futures){
   String result = future.get();
 
   //Compute the result
}
 
pool.shutdown();

But this code is a bit complicated. And there is a disadvantage. If the first task takes a long time to compute and all the other tasks ends before the first, the current thread cannot compute the result before the first task ends. Once again, Java has the solution for you, CompletionService.

A CompletionService is a service that make easier to wait for result of submitted task to an executor. The implementation is ExecutorCompletionService who’s based on an ExecutorService to work. So let’s try :

ExecutorService threadPool = Executors.newFixedThreadPool(4);
CompletionService<String> pool = new ExecutorCompletionService<String>(threadPool);
 
for(int i = 0; i < 10; i++){
   pool.submit(new StringTask());
}
 
for(int i = 0; i < 10; i++){
   String result = pool.take().get();
 
   //Compute the result
}
 
threadPool.shutdown();

With that, you have the result in the order they are completed and you don’t have to keep a collection of Future.

Here we are, you have the tools in hand to launch tasks in parralel using performing thread pools. Using Executors, ExecutorService and CompletionService you can create complex algorithm using several taks. With that tools, it’s really easy to change the number of threads performing in parralel or adding more tasks without changing a lot of code.

I hope that this post will help you to write better concurrent code.