Oracle SOA and Coherence

SSH prompting for password even after generating the RSA keys with empty password

2013-08-22T23:16:00.003-05:00

I struggled with this issue almost every time I wanted to login to remote hosts without entering the passwords. In some places, I used this solution because I was lazy and in some places it is a requirement such as, Hadoop setup. Anyways, the key to the solution is permissions and the steps are as under:

{USER}@machine1> rm -rf /home/{USER}/.ssh
{USER}@machine2> rm -rf /home/{USER}/.ssh /tmp/id_rsa.pub
{USER}@machine1> ssh-keygen -t rsa [Enter, Enter, Enter]
{USER}@machine1> scp /home/{USER}/.ssh/id_rsa.pub {USER}@machine2:/tmp
{USER}@machine2> mkdir /home/{USER}/.ssh
{USER}@machine2> cat /tmp/id_rsa.pub >> /home/{USER}/.ssh/authorized_keys
{USER}@machine2> chmod -R 700 /home/{USER}/.ssh
{USER}@machine1> chmod 600 /home/{USER}/.ssh/id_rsa*
{USER}@machine2> rm /tmp/id_rsa.pub
{USER}@machine1> chmod 755 /home/{USER}
{USER}@machine2> chmod 755 /home/{USER}

In above steps, replace the {USER} with your user and machine1 and machine2 with appropriate machine ip addresses.h

Please note, the steps marked in BOLD and they should have exactly the same permissions.

Step by Step guide to enable X11 forwarding using Cygwin

2013-08-21T17:52:00.000-05:00

I always found a need of X11 forwarding and so writing this step by step guide for reference:

Step 1: Install Cygwin/X: I would like to take you through the minimum required packages for Cygwin/X and here are the detailed steps:

Download the cygwin setup file from http://cygwin.com/install.html
Run the Cygwin setup program and you will see the welcome screen.Click Next to proceed to the next screen.
Choose, Install from Internet, this will still save the package files to your download directory so that you can install Cygwin on any number of machines. Click Next to proceed to the next screen.
The default Install Root is c:\cygwin which should be fine for most installations. Leave Default Text File Type as UNIX. Leave Install For set to All unless you lack local administrative privileges.
Click Next to proceed to the next screen.
Local Package Directory should default to the directory that you ran setup.exe from.Click Next to proceed to the next screen.
Choose your proxy setup, or, just choose Direct Connection if no proxy is needed.Click Next to proceed to the next screen.
Select the first mirror for downloading. Click Next to proceed to the next screen; setup will download a list of available packages as it moves to the next screen.
On the next screen you will select the packages that will be downloaded and installed. A listing of the Cygwin/X packages is given below; a listing of the general Cygwin packages would be beyond the scope of this document.
Cygwin/X packages are located in the X11 category:

xorg-server (required, the Cygwin/X X Server)
xinit (required, scripts for starting the X server: xinit, startx, startwin (and a shortcut on the Start Menu to run it),startxdmcp.bat )
xorg-docs (optional, man pages)
You may also select any X client programs you want to use, and any fonts you would like to have available.
You may also want to ensure that the openssh package is selected if you wish to use ssh connections to run remote X clients.
You may also want to ensure that the inetutils or rsh packages are selected if you wish to use telnet or rsh connections to run remote X clients. (not recommended)

Click Next to begin the download process, you may want to try another mirror if you see a "Connecting" message on this screen for a long period of time.
You have now successfully installed Cygwin/X.

Step 2: On your desktop, start the XWin Server from Start > All Programs > Cygwin-X > XWin Server

Step 3: On the XWin Server window, type the following:

ssh -X -C user@hostname
type xclock and it should throw the xclock display

Step 4: If you need to sudo to a different user and still make the X11 work, you need to do the following (before running the sudo command):

xauth list

hostname/unix:13 MIT-MAGIC-COOKIE-1 ada9344e4a990d3b05d3bf66a9948758

hostname/unix:10 MIT-MAGIC-COOKIE-1 11b30b1d65d90e64d15d811a97b9fb20

hostname/unix:11 MIT-MAGIC-COOKIE-1 ddaa0d9fc7da0ee5228aa459ccdf427c

sudo su - <>
echo $DISPLAY

localhost:11.0

xauth add hostname/unix:11 MIT-MAGIC-COOKIE-1 ddaa0d9fc7da0ee5228aa459ccdf427c

11 in the $DISPLAY matches the 11 in the list of the xauth.

Now you should be able to throw the display from the remote machine to your local machine.

Takes too long to restart the Weblogic Managed Servers

2013-06-04T13:46:00.001-05:00

If your Weblogic server startup takes too long, then you might be running into entropy bug that can be fixed in one of the 3 ways:

Patch /jre/lib/security/java.security by replacing the line "securerandom.source=file:/dev/urandom" with "securerandom.source=file:/dev/./urandom"
Add JVM argument -Djava.security.egd=file:/dev/./urandom to the startup scripts
Create a symlink for /dev/urandom to point to /dev/./urandom

Updating a large table in batches

2013-06-03T13:09:00.000-05:00

In order to update million of record using the UPDATE statement can be time-consuming and may lead to timeout errors and other issues. A better approach would be to commit the updates in batches and here is the PL/SQL recipe for doing it:

declare
cursor c is select rowid id from <TABLE> order by rowid;
counter NUMBER(10) := 0;
rowids dbms_sql.urowid_table;

begin
dbms_output.enable;
open c;
loop
fetch c bulk collect into rowids limit 1000;
forall i in 1..rowids.count
update <TABLE> set date=SYSDATE
where rowid = rowids(i);
commit;
counter := counter + 1;
dbms_output.put_line('Updated batch ' || counter );
exit when c%notfound;
end loop;
close c;
end;
/

The above PL/SQL would update the DATE column of the table <TABLE> in batches of 1000 and will be much faster than the traditional UPDATE statement.

How can you timeout invocation of external endpoints from BPEL

2012-10-25T15:24:00.000-05:00

SyncMaxWaitTime setting applies to synchronous process invocations when the process has a breakpoint in the middle of the process. If there are no breakpoints, the entire process will be executed by the client thread. If there is a breakpoint then a new thread will be spawned to continue the processing after the break. For more details, follow the link.

In order to explicitly set the timeout for the endpoints invoked from within the BPEL, use the following reference binding properties to configure timeouts while invoking external services.

<reference name="HWService">
<interface.wsdl interface="writeHW_ptt">
<binding.ws port="helloWS">
<property name="oracle.webservices.httpReadTimeout" type="xs:string" many="false">10000</property>
<property name="oracle.webservices.httpConnTimeout" type="xs:string" many="false">10000</property>
</binding.ws>
</reference>

The property "oracle.webservices.httpReadTimeout" specifies how long to wait until the target service processes the request and "oracle.webservices.httpConnTimeout" specifies the wait-time to connect to the external service.

In asynchronous invocations, you may use the Pick action for configuring the invocation timeouts.

Oracle Traffic Director : Extract Private Key to Decrypt and View SSL Snoop Data

2012-10-19T18:08:00.001-05:00

Oracle Traffic Director (OTD) is the last software load balancer released and is based on iPlanet Web Server. It is a fast, reliable, and scalable layer-7 software load balancer that you can deploy as the reliable entry point for all HTTP and HTTPS traffic to application servers and web servers in your network. It leverage the NSS Shared DB for storing the private key and certificates for the SSL encryption and if you are looking to decrypt the SSL traffic using the private key, you would require to first extract it and the steps for it are as under:

For some reason, the pk12util that comes with OTD installation did not work for me so I have to move the cert9.db and key4.db onto my windows machine and follow the below steps:

Downloaded NSS Tools for windows from here: NSS_Tools_x86_from_NSS_3.12.7 Tools.zip into C:\
Copied the key4.db and cert9.db to “C:\Users\nj\keys” folder
Go to command prompt (cmd C: ) and executue c:\pk12util.exe -o C:\Users\nj\keys\cert.p12 -d sql:C:\Users\nj\keys -n "<>" (populated on SSL >> Server Certificates on OTD Admin Console)
Prompted for password, enter <>
This should create a cert.p12 under keys folder
Use OpenSSL to execute: openssl pkcs12 –in cert.p12 –out private.key –nocerts –nodes
Prompted for password, enter <>
private.key file should be created in the folder

Cisco VPN error : The VPN Client was unable to setup IP Filtering

2012-10-18T16:24:00.001-05:00

If you are getting an error "The VPN Client was unable to setup IP Filtering" when trying to use the Cisco Any Connect client then here is the solution for you:

Save the file "BFE.reg" locally and then execute the file by double clicking
Click Start > Run > regedit
Browse to “HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\BFE\Parameters\Policy”
Right click on “Policy” and select permission
In the "Permissions for Policy" window, select advanced
Unselect “Include inheritable permission from this object’s parent”
Select Add from the Windows Security popup box
Remove Users and CREATOR OWNER

Select Add button
Enter in "NT Service\BFE" and select OK
Give the Object the following Allow permissions: Query Value, Set Value, Create Subkey, Enumerate Subkeys, Notify, and Read Control
Select OK to close all of the boxes

Reboot Windows
Connect with AnyConnect to test the connection

Weblogic setting wrong protocol in WSDL (Load Balancer terminating SSL)

2012-10-18T12:56:00.000-05:00

In most of the architectures, the SSL is terminated at the hardware load balancer for performance reasons and allows the internal traffic to use HTTP for communication.

Client ---[HTTPS] --> Hardware LB (SSL termination) --- [HTTP] --> WLS (WebService)

The client will typically fetch the WSDL for the webservice hosted on WLS and use the endpoint available in the WSDL for invoking the webservice. So the calls for fetching the WSDL would happen as under:

Client (https://lbhost:lbhttpsport/URI?wsdl) --- > Hardware LB (http://wlshost:wlshttpport?wsdl) --- WLS (set the endpoint in wsdl as frontendhost:frontendhttpport if provided or will return http://lbhost:lbhttpsport/URI)

Please note that the endpoint in the WSDL has the http protcol whereas the client is only used to call the LB on https protocol. The reason why WLS sets the protocol as http is because the request was recieved on http and there is no way for WLS to identify if the actual request was made on https.

To solve the issue, you need to set an extra header variable "WL-Proxy-SSL: true" at the load balancer so that WLS identifies the request is called on https. Also, you need to set the flag Weblogic-Proxy-Plugin Enabled at the WLS managed server.

Weblogic Service Migration (Issues and Workarounds)

2012-10-18T12:04:00.002-05:00

Pinned services, such as JMS-related services, the JTA Transaction Recovery Service, and user-defined singleton services are hosted on individual server instances within a cluster—for these services, the WebLogic Server supports failure recovery with service migration. There is a lot of documentation and blogging on this topic and in this post I want to just cover two of the issues that you may face during the service migration setup:

Issue1: If you have multiple clusters within a domain and you have setup service migration (database leasing) for only some of the clusters in your domain then you may find that the other cluster members start throwing errors as under:

" #### <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <> <> <8bca0b730cda1738:17608ce6:1382c32b7b9: -8000-0000000000000002="-8000-0000000000000002"> <1340771329481> 'WseeJmsModule'.

java.lang.IllegalArgumentException: Cannot add Singleton Service M_MS1 (migratable) as SingletonServicesManager not started. Check if MigrationBasis for cluster is configured."

Workaround: Configure Database leasing for all the clusters in the domain though no need to configure the full service migration but just the cluster level service migration.

Issue2: If you are using multi-datasource (MDS) for your service migration then you may see that everything is working fine but actually behind the scenes the service migration framework pins itself to first datasource in MDS list and it really does not failover to the other datasource in case if the first datasource goes DOWN. You can easily find if this is the case by issuing the following database query:

" select username, gv$sqlarea.inst_id, sql_text, gv$sqlarea.executions, gv$sqlarea.first_load_time from gv$session, gv$sqlarea where gv$session.sql_id = gv$sqlarea.sql_id and username ='<db_username>'; "

You see that all the sql to udpate the ACTIVE table are issued against the same RAC instance/datasource. If you shutdown the RAC instance/datasource where the service migration framework is pinned, it results that no more update happen to the ACTIVE table. Please note that the server periodically renews its lease by updating the timestamp in the lease table. By default a migratable server renews its lease every 30,000 milliseconds—the product of two configurable ServerMBean properties:

HealthCheckIntervalMillis, which by default is 10,000.
HealthCheckPeriodsUntilFencing, which by default is 3.

But there will no session created once the first datasource in the MDS configuration goes DOWN. Though, there will be lot of exceptions in the managed server logs but neither the migration happens nor the managed are able to secure a lease.

Workaround: There are couple of workarounds to resolve the issue:

1) Use TNS connect string for the datasource rather than using MDS

2) This is a reported bug (9365773) and should ask the Oracle support for a patch to fix the issue.

Also would like to mention some of the debug parameters specific for logging the service migration internals as under:

-Dweblogic.StdoutDebugEnabled=true

-Dweblogic.log.LoggerSeverity=Debug

-Dweblogic.log.LogSeverity=Debug

-Dweblogic.debug.DebugServerMigration=true

-Dweblogic.debug.DebugSingletonServices=true

-Dweblogic.debug.DebugUnicastMessaging=true

-Dweblogic.debug.DebugServerLifeCycle=true

-Dweblogic.slcruntime=true

-Dweblogic.slc=true

Please note that, both the above issue happened in Weblogic 10.3.4 & 10.3.5 and might have got fixed in later versions of Weblogic releases.

What is T3, soadirect, sb? When to use them?

2012-07-17T19:34:00.000-05:00

Weblogic installation provides a range of standalone client that access WebLogic Server applications range from simple command-line utilities that use standard I/O to highly interactive GUI applications built using the Java Swing/AWT classes.

T3: RMI communications in WebLogic Server use the T3 protocol to transport data between WebLogic Server and other Java programs, including clients and other WebLogic Server instances. A server instance keeps track of each Java Virtual Machine (JVM) with which it connects, and creates a single T3 connection to carry all traffic for a JVM.

SOADIRECT: The SOADIRECT transport provides native connectivity between Oracle Service Bus and Oracle SOA Suite service components. Oracle SOA Suite provides a "direct binding" framework that lets you expose Oracle SOA Suite service components in a composite application, and the Oracle Service Bus SOA-DIRECT transport interacts with those exposed services through the SOA direct binding framework, letting those service components interact in the service bus layer and leverage the capabilities and features of Oracle Service Bus.

SB Transport: SOA Suite can invoke Oracle Service Bus proxy services with an SB transport binding, including the transaction and security context using the direct binding reference.

You can google for more details on each of them. Now, all of the above internally uses RMI/t3 for protocol for communication But, I could not find any specific internals about how t3 works and most importantly how to load balance these using software/hardware load balancer. In this blog, I will reveal the details that I got from the experts:

1. Can you loadbalance t3 connections?

You cannot explicitly load balance t3 connections. So, even if you put a hardware/software loadbalancer in front of SOA/OSB/Weblogic cluster it would not help. Yes, the initial connection request will go through the load balancer (only as part of connection establishment i.e. as an alternative to DNS round-robin) but after the intial connection is established, it will stay for the lifetime of the server or client. Once a connection(t3) is established, it consistently goes routes back to the same server directly without going through the loadbalancer. If the connection fails, the loadbalancer doesn't attempt to re-establish it. Responsibility for failover and reconnection remains with the t3 protocol itself.

2. What happens when the client/SOA/OSB requests a t3/soadirect/sb connection?

In general, t3 takes care of its own load balancing (stateless stubs will round-robin calls; stateful stubs will be sticky, with appropriate failover), but requires unfettered access to establish connections to the target servers. Further, once a client has made a connection to one node in the cluster, it learns about all cluster members (regardless of the contents of the URL used to connect); and is actively notified of the member health so it will rarely try to contact a server that is not running. T3 is a stateful protocol; once a connection has been established, the LB cannot re-route it. Responsibility for failover and reconnection must remain with the RMI stub.

3. What happens if the connection is dropped?

Once the channel (t3) is established, the two endpoints will issue regular heartbeats and keep the connection alive.

4. how can you loadbalance your t3 connections?

No, you cannot loadbalance them so it is recommended to not use it until you need to propogate transactions as t3 provides transactional capabilities. You may some link specifying that soadirect/sb/t3 is faster than regular http/ws but I would recommend don't believe it until you test it in your environment.

Bottom line: Use sb/soadirect/t3 only if you need to propogate transactions between end systems else don't use it!!

SOA, OSB and Weblogic Performance Tuning

2012-07-10T11:28:00.000-05:00

I have compiled this document with references to various links/documents/guides illustrating performance tuning for SOA, OSB and Weblogic infrastructure:

Performance Tuning Guide
[http://docs.oracle.com/cd/E23943_01/core.1111/e10108/toc.htm]
Tuning Your SOA Infrastructure for Performance and Scalability OOW 2011[http://www.oracle.com/technetwork/middleware/soasuite/learnmore/odtugsoaperformancetuningsession-427186.pdf]
SOA Suite Performance Tuning Presentation OOW 2010 [http://www.oracle.com/technetwork/middleware/soasuite/soaperformancetuning-176286.pdf]

[http://www.oracle.com/technetwork/database/features/availability/soa11gstrategy-1508335.pdf]

Oracle SOA Suite 11g WP on DB-RAC Configuration [http://www.oracle.com/technetwork/database/focus-areas/availability/maa-fmw-soa-racanalysis-427647.pdf]

JVM Memory Monitoring, Tuning, Garbage Collection, Out of Memory, and Heap Dump Analysis For SOA Suite Integration 11g (Doc ID 1358719.1)

Oracle Fusion Middleware Performance and Tuning for Oracle WebLogic Server 11g [http://download.oracle.com/docs/cd/E21764_01/web.1111/e13814.pdf]

Oracle Fusion Middleware Performance and Tuning Guide (This one include SOA and OSB) [http://download.oracle.com/docs/cd/E21764_01/core.1111/e10108.pdf]

Oracle Service Registry Installation Issue (Error 500 Internal Service Error)

2012-07-06T16:33:00.000-05:00

If you are trying to install OSR 11g in a domain that has OSB or SOA managed servers configured, you will run into one of the issues as below:

java.lang.LinkageError: Class javax/xml/namespace/QName violates loader constraints
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClassCond(ClassLoader.java:631)
at java.lang.ClassLoader.defineClass(ClassLoader.java:615)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)
at weblogic.utils.classloaders.GenericClassLoader.defineClass(GenericClassLoader.java:343)
Truncated. see log file for complete stacktrace

org.idoox.wasp.WaspInternalException: java.lang.RuntimeException: Updates to config files not supported
at com.systinet.wasp.WaspImpl.boot(WaspImpl.java:399)
at org.systinet.wasp.Wasp.init(Wasp.java:151)
at com.systinet.transport.servlet.server.Servlet.init(Unknown Source)
at weblogic.servlet.internal.StubSecurityHelper$ServletInitAction.run(StubSecurityHelper.java:283)
at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
Truncated. see log file for complete stacktrace
Caused By: java.lang.RuntimeException: Updates to config files not supported
at com.idoox.config.xml.XMLConfigurator.updateConfigFile(XMLConfigurator.java:720)
at com.idoox.config.xml.XMLConfigurator.prepareConfigFile(XMLConfigurator.java:570)
at com.idoox.config.xml.XMLConfigurator.init(XMLConfigurator.java:423)
at org.idoox.config.Configurator.init(Configurator.java:65)
at com.systinet.wasp.WaspImpl.boot(WaspImpl.java:359)
Truncated. see log file for complete stacktrace

The reason for this issue is the conflict among the classpath jars and the solution is as below:
Replace the snippet in setDomainEnv.sh/cmd where POST_CLASSPATH variable is created as below (sample from SOA +OSR and same applies to OSB+OSR):

if NOT "%SERVER_NAME%"=="osr_server1" (
set POST_CLASSPATH=%SOA_ORACLE_HOME%\soa\modules\oracle.soa.fabric_11.1.1\oracle.soa.fabric.jar;%SOA_ORACLE_HOME%\soa\modules\oracle.soa.fabric_11.1.1\fabric-runtime-ext-wls.jar;%SOA_ORACLE_HOME%\soa\modules\oracle.soa.adapter_11.1.1\oracle.soa.adapter.jar;%SOA_ORACLE_HOME%\soa\modules\oracle.soa.b2b_11.1.1\oracle.soa.b2b.jar;%POST_CLASSPATH%
set POST_CLASSPATH=%DOMAIN_HOME%\config\soa-infra;%SOA_ORACLE_HOME%\soa\modules\fabric-url-handler_11.1.1.jar;%SOA_ORACLE_HOME%\soa\modules\quartz-all-1.6.5.jar;%POST_CLASSPATH%
set POST_CLASSPATH=%COMMON_COMPONENTS_HOME%\modules\oracle.xdk_11.1.0\xsu12.jar;%BEA_HOME%\modules\features\weblogic.server.modules.xquery_10.3.1.0.jar;%SOA_ORACLE_HOME%\soa\modules\db2jcc4.jar;%POST_CLASSPATH%
set POST_CLASSPATH=%UMS_ORACLE_HOME%\communications\modules\usermessaging-config_11.1.1.jar;%POST_CLASSPATH%
set POST_CLASSPATH=D:\SOA\installations\11g6\tempInstallation\Oracle_SOA1\soa\modules\oracle.soa.common.adapters_11.1.1\oracle.soa.common.adapters.jar;%POST_CLASSPATH%
set POST_CLASSPATH=%COMMON_COMPONENTS_HOME%\soa\modules\commons-cli-1.1.jar;%COMMON_COMPONENTS_HOME%\soa\modules\oracle.soa.mgmt_11.1.1\soa-infra-mgmt.jar;%POST_CLASSPATH%
) else (
set POST_CLASSPATH=""
)

Basically, you need to remove all the jars from the POST_CLASSPATH for the OSR managed server. This is bug [Patch 9499508] as reported on oracle support.

Managing Database growth: SOA 11g Running Purge Scripts -> A Step by Step Approach

2012-06-20T22:22:00.002-05:00

Recently, there is a step by step document has been published for using the purge scripts bundled with SOA 11.1.1.4 + versions which can be found here [ID 1345957.1].

Also, the state details have been really confusing and the details but a consolidated document can be found here [ID 1362028.1].

Best Practices for Coherence Portable Object Format (POF)

2012-06-04T10:15:00.002-05:00

POF objects are indexed so it is possible to quickly traverse/navigate the binary (serialized form) to a specific element for extraction or updating using the indices without de-serializing the entire binary. Out of the box, coherence provides SimplePofPath class that can navigate a POF value based on integer indices. In the simplest form, all you need to do is to provide the index of the attribute that you want to extract/update. Some of the best practices that are coming to my mind are as under:

Order your reads and writes: start with the lowest index value in the serialization routine and finish with the highest. When deserializing a value, perform reads in the same order as writes.

Use the smallest possible integer for indexing the objects

Non-contiguous indexes are acceptable but must be read/written sequentially.

For subclasses reserve index ranges: index's are cumulative across derived types. As such, each derived type must be aware of the POF index range reserved by its super class.

Do not re-purpose indexes: to support Evolvable, it's imperative that indexes of attributes are not re-purposed across class revisions.

Label each of the index for the attribute

The most frequently used attribute in the Filter(s) and Extractor(s) should be assigned lowest index.

Enable POF Object reference if the objects have circular/nested references (feature available in 3.7.1 + only)

Coherence Extend vs TCMP member

2012-05-03T12:46:00.001-05:00

Coherence* Extend provides lot more operation flexibility at the cost of slight performance because the communication protocol changes from TCMP to TCP and an extra hop. Here are some of the advantages of using C*Extend:

- Client application and coherence servers can run different coherence versions and can be upgraded independently
- Client and server configuration can be managed independently
- Data Access can be secured more efficiently
- Client application and servers can reside across subnets and WANs

Some of the disadvantages of using C*Extend:
- Extra hop that occurs through the proxy
- Extra JVM and hardware to run and manage proxies including, load balancing proxies, memory allocation for proxies, number of proxies and so on.
- Increased latency due to an extra hop and use of TCP compared to TCMP.

I would recommend to use C*Extend if:

1) client application(s) may join and/or leave the coherence cluster frequently (common reason is client application performing frequent long GCs)
2) client application(s) does not reside in the same subnet as the coherence servers
3) client application(s) are not tightly bound at an application layer to the coherence cluster.
4) client application(s) and coherence servers are managed by two different groups and may have have different upgrade schedules.
5) Requires data access security between client application(s) and coherence servers.

Installation Issues : Oracle Enterprise Repository 11.1.1.5.0

2011-10-27T19:41:00.000-05:00

During the configuration of the OER domain, you might get an error as below:

2011-10-18 09:42:45,796 INFO [WizardController] com.oracle.cie.domain.template.catalog.impl.ComponentsXMLConverter - /u01/app/oracle/middleware/wlserver_10.3/common/lib/components.xml does not contain component elements and will be skipped
2011-10-18 09:42:45,816 INFO [WizardController] com.oracle.cie.wizard.domain.gui.tasks.MultipleTemplateSelectionGUITask - Selected required template: Basic WebLogic Server Domain:10.3.4.0 [/u01/app/oracle/middleware/wlserver_10.3/common/templates/domains/wls.jar]
2011-10-18 09:43:21,132 INFO [WizardController] com.oracle.cie.domain.WLSTemplateBuilder - No config-groups.xml found in template
2011-10-18 09:43:21,133 INFO [WizardController] com.oracle.cie.domain.WLSTemplateBuilder - No template-database.xml found in template
2011-10-18 09:43:21,584 INFO [WizardController] com.oracle.cie.domain.AbstractTemplate - Adding top-level component reference: WebLogic Server version 10.3.5.0
2011-10-18 09:43:21,589 INFO [WizardController] com.oracle.cie.domain.AbstractTemplate - Setting symbol for component WebLogic Server version 10.3.5.0 to WebLogic_Server_10.3.5.0_wlserver_10.3_ORACLE_HOME
2011-10-18 09:43:21,711 INFO [WizardController] com.oracle.cie.domain.WLSTemplateBuilder - No template-database.xml found in template
2011-10-18 09:43:21,752 WARN [WizardController] com.oracle.cie.domain.AbstractTemplate - Could not find component name 'Oracle Enterprise Repository' version 'null'
2011-10-18 09:43:21,753 INFO [WizardController] com.oracle.cie.domain.AbstractTemplate - Replacing component reference Oracle Enterprise Repository version null with top-level component reference: WebLogic Server version 10.3.5.0
2011-10-18 09:43:21,753 INFO [WizardController] com.oracle.cie.domain.AbstractTemplate - Setting symbol for component WebLogic Server version 10.3.5.0 to WebLogic_Server_10.3.5.0_wlserver_10.3_ORACLE_HOME
2011-10-18 09:43:21,775 WARN [WizardController] com.oracle.cie.domain.aspect.XBeanConfigAspect - Unable to locate property:Target on type:class com.oracle.cie.domain.xml.configxb.impl.MachineTypeImpl
2011-10-18 09:43:21,783 WARN [WizardController] com.oracle.cie.domain.aspect.XBeanConfigAspect - Unable to locate property:Target on type:class com.oracle.cie.domain.xml.configxb.impl.ServerTypeImpl
2011-10-18 09:44:01,986 ERROR [WizardController] com.oracle.cie.common.util.JdkVersion - JDK version is null.
2011-10-18 09:46:37,077 INFO [WizardController] com.oracle.cie.domain.assignment.ServerMachineAssignment - ListenPort internal Validation result= [null]
2011-10-18 09:46:54,552 INFO [WizardController] com.oracle.cie.domain.assignment.ServerMachineAssignment - ListenPort internal Validation result= [null]
2011-10-18 09:48:19,441 INFO [WizardController] com.oracle.cie.domain.assignment.ServerMachineAssignment - ListenPort internal Validation result= [null]
2011-10-18 09:48:21,707 INFO [WizardController] com.oracle.cie.domain.DomainChecker - ListenPort external Validation result= [null]
2011-10-18 09:48:26,582 ERROR [create_gui] com.oracle.cie.wizard.domain.gui.tasks.DomainCreationGUITask - Generation Error!!
Traceback (innermost last):
(no code object) at line 0
SyntaxError: ('Lexical error at line 6, column 11. Encountered: "@" (64), after : ""', ('', 6, 11, 'oerPort = @DOMAIN_LISTEN_PORT@'))

at org.python.core.parser.fixParseError(Unknown Source)
at org.python.core.parser.parse(Unknown Source)
at org.python.core.Py.compile_flags(Unknown Source)
at org.python.util.PythonInterpreter.execfile(Unknown Source)
at org.python.util.PythonInterpreter.execfile(Unknown Source)
at com.oracle.cie.domain.script.ScriptHelper.executeEmbeddedTemplateScript(ScriptHelper.java:1571)
at com.oracle.cie.domain.DomainChecker.executeEmbeddedScript(DomainChecker.java:1322)
at com.oracle.cie.domain.DomainChecker.runCustomWLSTScript(DomainChecker.java:1305)
at com.oracle.cie.domain.DomainChecker.preprocessOutput(DomainChecker.java:143)
at com.oracle.cie.domain.DomainGenerator.generate(DomainGenerator.java:269)
at com.oracle.cie.wizard.domain.gui.tasks.DomainCreationGUITask$1.run(DomainCreationGUITask.java:216)

The reason for the issue is that the OER is actually not properly installed even though the installer specifies that the installation is successfully completed. You can check this by mentioning the log file during the installation as below:

java -jar OER111150_generic.jar -log=\tmp\install.log

You will see the error "Cannot set up jar file extraction directory: Permission denied" in the log file. The reason for this error is that the installer extracts itself in the /tmp directory on the file system where the installer resides (if installer is on shared server then the installer tries to extracts itself on the shared server /tmp directory). Give full permissions to the user on the /tmp directory of the server where the installer resides.

Also specify the environment variables as below:

export TMP=/tmp
export TMPDIR=/tmp

In windows, specify the environment variable TEMP=C:\TEMP and TEMPDIR=C:\TEMP

Oracle SOA 11g : Securing and Calling Secured Web Services (Example)

2011-10-27T18:51:00.000-05:00

In Oracle Fusion Middleware 11g , Oracle Web Services Manager (WSM) security and management has been completely redesigned and rearchitected. The previous release, Oracle WSM 10g, was delivered as a standalone product or as a component of the Oracle SOA Suite. In the 11g release, Oracle WSM has been integrated with Oracle WebLogic Server as part of the Oracle Fusion Middleware SOA Suite.

In this post, I would explain how WSM integration can be used to secure a composite and also how to invoke a secured composite/ webservice. In order to accomplish it, let's create 2 composite projects:

- Child Process (secured composite): This composite has a simple BPEL project that accepts a string and returns the same string with "Hello " appended.

- Parent Process (unsecured composite calling the Child Process): This composite has a BPEL process that accepts the string, invokes the Child Process passing the input string and returns the output from the Child Process back to the calling interface.

I am skipping the development of the child and parent processes/composites as it is very simple and will detail how to configure Policies as below:

- Child Process:

Step1: Go to composite.xml and right click on the Exposed Services and click "Configure WS Policies" as below:

Step2: You can configure the policy pipeline but for the sake of simplicity, I will only configure the Security using the "oracle/wss_username_token_service_policy" as shown below:

Step3: Deploy the composite and test it using the em console. Remember now the composite is secured and requires a username token as part of the SOAP Header as below:

<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">
<soap:Header>
<wsse:Security soap:mustUnderstand="1"
xmlns:wsse="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-secext-1.0.xsd">
<wsse:UsernameToken xmlns:wsse="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-secext-1.0.xsd"
wsu:Id="UsernameToken-FvmfJSSMf71Q3Z9gn4chiQ22"
xmlns:wsu="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-utility-1.0.xsd">
<wsse:Username>weblogic</wsse:Username>
<wsse:Password Type="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-username-token-profile-1.0#PasswordText">welcome1</wsse:Password>
</wsse:UsernameToken>
</wsse:Security>
</soap:Header>
<soap:Body xmlns:ns1="http://xmlns.oracle.com/SecurityApplication_jws/ChildProcess/SecuredSvc">
<ns1:process>
<ns1:input>Neeraj</ns1:input>
</ns1:process>
</soap:Body>
</soap:Envelope>

Either modify the SOAP packet as above in the XML View or use the WSS UserName Token under Security configuration as below:

Parent Process:

Step 1: Go to composite.xml and right click on the External References (Child Process) and click on "Configure WS Policies". Select the "oracle/wss_username_token_client_policy" policy under security.

Step 2: Open the composite.xml in Source view and add the following properties under <reference> tag as below:

<reference name="SecuredBPEL"
:
:
<wsp:PolicyReference URI="oracle/wss_username_token_client_policy"
orawsp:category="security" orawsp:status="enabled"/>
<property name="weblogic.wsee.wsat.transaction.flowOption"
type="xs:string" many="false">WSDLDriven</property>
<property name="oracle.webservices.auth.username" type="xs:string"
many="false" override="may">weblogic</property>
<property name="oracle.webservices.auth.password" type="xs:string"
many="false" override="may">welcome1</property>
</binding.ws>
</reference>

Make sure the username and password is same as configured in the Weblogic Server.

Step3: Test the Parent process and it should be able to invoke the secured child process.

Few things to note:

1. All the above policy configurations can be done at runtime using the EM console without doing anything at the design time.

2. In the above example, we have hard-coded the username/password at development which should be made configurable as mentioned below:

- Go to EM console and right click on ParentProcess ->Service/Reference Properties -> PartnerLink -> Enter the username and password for the Child service in the below screen

Oracle SOA 11g : Configuring WS Policies

2011-10-27T13:28:00.000-05:00

In Oracle 11g, OWSM has been integrated with Oracle SOA Suite and now you can configure the policies at design time (using Jdeveloper in composite.xml) or at runtime (using EM console). The re-architecture of OWSM is detailed here.

- At Design time: Go to composite.xml (Design View) and click on either Exposed Service or External References to configure policies on this composite or referenced services respectively.

- At Runtime: Go to em console, right click on the process, select interface or reference where the policy should be applied.

Click on policies tab and attach the appropriate policies that needs to be applied.

Coherence and Database Integration Examples

2011-10-11T14:38:00.001-05:00

Troubleshooting : Coherence Log Message

2011-10-06T14:40:00.000-05:00

Configuration Log Messages

Message	java.io.IOException: Configuration file is missing: "tangosol-coherence.xml"
Parameters	n/a
Severity	1-Error
Cause	The operational configuration descriptor cannot be loaded.
Action	Make sure that the "tangosol-coherence.xml" resource can be loaded from the class path specified in the Java command line.

Message	Loaded operational configuration from resource "%s"
Parameters	%s - the full resource path (URI) of the operational configuration descriptor
Severity	3-Informational
Cause	The operational configuration descriptor is loaded by Coherence from the specified location.
Action	If the location of the operational configuration descriptor was explicitly specified via system properties or programmatically, verify that the reported URI matches the expected location.

Message	Loaded operational overrides from "%s"
Parameters	%s - the URI (file or resource) of the operational configuration descriptor override
Severity	3-Informational
Cause	The operational configuration descriptor points to an override location, from which the descriptor override has been loaded.
Action	If the location of the operational configuration descriptor was explicitly specified via system properties, descriptor override or programmatically, verify that the reported URI matches the expected location.

Message	Optional configuration override "%s" is not specified
Parameters	%s - the URI of the operational configuration descriptor override
Severity	3-Informational
Cause	The operational configuration descriptor points to an override location which does not contain any resource
Action	Verify, that the operational configuration descriptor override is not supposed to exist

Message	java.io.IOException: Document "%s1" is cyclically referenced by the 'xml-override' attribute of element %s2
Parameters	%s1 - the URI of the operational configuration descriptor or override; %s2 - the name of the XML element that contains an incorrect reference URI
Severity	1-Error
Cause	The operational configuration override points to itself or another override that point to it, creating an infinite recursion
Action	Correct the invalid 'xml-override' attribute's value.

Message	java.io.IOException: Exception occurred during parsing: %s
Parameters	%s - the XML parser error
Severity	1-Error
Cause	The specified XML is invalid and cannot be parsed.
Action	Correct the XML document.

Message	Loaded cache configuration from "%s"
Parameters	%s - the URI (file or resource) of the cache configuration descriptor
Severity	3-Informational
Cause	The operational configuration descriptor or a programmatically created ConfigurableCacheFactory instance points to a cache configuration descriptor that has been loaded.
Action	Verify that the reported URI matches the expected cache configuration descriptor location

Partitioned Cache Service Log Messages

Message	Asking member %n1 for %n2 primary partitions
Parameters	%n1 - the node id this node asks to transfer partitions from; %n2 - the number of partitions this node is willing to take
Severity	4-Debug Level 4
Cause	When a storage-enabled partitioned service starts on a Coherence node, it first receives the configuration update that informs it about other storage-enabled service nodes and the current partition ownership information. That information allows it to calculate the "fair share" of partitions that each node is supposed to own at the end of the re-distribution process. This message demarcates a beginning of the transfer request to a specified node for a number of partitions to move toward the "fair" ownership distribution.
Action	None.

Message	Transferring %n1 out of %n2 primary partitions to member %n3 requesting %n4
Parameters	%n1 - the number of primary partitions this node transferring to a requesting node; %n2 - the total number of primary partitions this node currently owns; %n3 - the node id that this transfer is for; %n4 - the number of partitions that the requesting node asked for
Severity	4-Debug Level 4
Cause	During the partition distribution protocol, a node that owns less than a "fair share" of primary partitions requests any of the nodes that own more than the fair share to transfer a portion of owned partitions. The owner may choose to send any number of partitions less or equal to the requested amount. This message demarcates the beginning of the corresponding primary data transfer.
Action	None.

Message	Transferring %n1 out of %n2 partitions to a machine-safe backup 1 at member %n3 (under %n4)
Parameters	%n1 - the number of backup partitions this node transferring to a different node; %n2 - the total number of partitions this node currently owns that are "endangered" (do not have a backup); %n3 - the node id that this transfer is for; %n4 - the number of partitions that the transferee can take before reaching the "fair share" amount
Severity	4-Debug Level 4
Cause	After the primary partition ownership is completed, nodes start distributing the backups, ensuring the "strong backup" policy, that places backup ownership to nodes running on machines that are different from the primary owners' machines. This message demarcates the beginning of the corresponding backup data transfer.
Action	None.

Message	Transferring backup%n1] for partition %n2 from member %n3 to member %n4
Parameters	%n1 - the index of the backup partition that this node transferring to a different node; %n2 - the partition number that is being transferred; %n3 the node id of the previous owner of this backup partition; %n4 the node id that the backup partition is being transferred to.
Severity	5-Debug Level 5
Cause	During the partition distribution protocol, a node that determines that a backup owner for one of its primary partitions is overloaded may choose to transfer the backup ownership to another, underloaded node. This message demarcates the beginning of the corresponding backup data transfer.
Action	None.

Message	Failed backup transfer for partition %n1 to member %n2; restoring owner from: %n2 to: %n3
Parameters	%n1 the partition number for which a backup transfer was in-progress; %n2 the node id that the backup partition was being transferred to; %n3 the node id of the previous backup owner of the partition
Severity	4-Debug Level 4
Cause	This node was in the process of transferring a backup partition to a new backup owner when that node left the service. This node is restoring the backup ownership to the previous backup owner.
Action	None.

Message	Deferring the distribution due to %n1 pending configuration updates
Parameters	%n1
Severity	5-Debug Level 5
Cause	This node is in the process of updating the global ownership map (notifying other nodes about ownership changes) when the periodic scheduled distribution check takes place. Before the previous ownership changes (most likely due to a previously completed transfer) are finalized and acknowledged by the other service members, this node will postpone subsequent scheduled distribution checks.
Action	None.

Message	Limiting primary transfer to %n1 KB (%n2 partitions)
Parameters	%n1 - the size in KB of the transfer that was limited; %n2 the number of partitions that were transfered
Severity	4-Debug Level 4
Cause	When a node receives a request for some number of primary partitions from an underloaded node, it may transfer any number of partitions (up to the requested amount) to the requestor. The size of the transfer is limited by the configuration element. This message indicates that the distribution algorithm limited the transfer to the specified number of partitions due to the transfer-threshold.
Action	None.

Message	DistributionRequest was rejected because the receiver was busy. Next retry in %n1 ms
Parameters	%n1 - the time in milliseconds before the next distribution check will be scheduled
Severity	6-Debug Level 6
Cause	This (underloaded) node issued a distribution request to another node asking for one or more partitions to be transferred. However, the other node declined to initiate the transfer as it was in the process of completing a previous transfer with a different node. This node will wait at least the specified amount of time (to allow time for the previous transfer to complete) before the next distribution check.
Action	None.

Message	Restored from backup %n1 partitions
Parameters	%n1 - the number of partitions being restored
Severity	3-Informational
Cause	The primary owner for some backup partitions owned by this node has left the service. This node is restoring those partitions from backup storage (assuming primary ownership). This message is followed by a list of the partitions that are being restored.
Action	None.

Message	Re-publishing the ownership for partition %n1 (%n2)
Parameters	%n1 the partition number whose ownership is being re-published; %n2 the node id of the primary partition owner, or 0 if the partition is orphaned
Severity	4-Debug Level 4
Cause	This node is in the process of transferring a partition to another node when a service membership change occurred, necessitating redistribution. This message indicates this node re-publishing the ownership information for the partition whose transfer is in-progress.
Action	None.

Message	%n1> Ownership conflict for partition %n2 with member %n3 (%n4!=%n5)
Parameters	%n1 - the number of attempts made to resolve the ownership conflict; %n2 - the partition whose ownership is in dispute; %n3 - the node id of the service member in disagreement about the partition ownership; %n4 - the node id of the partition's primary owner in this node's ownership map; %n5 - the node id of the partition's primary owner in the other node's ownership map
Cause	If a service membership change occurs while the partition ownership is in-flux, it is possible for the ownership to become transiently out-of-sync and require reconciliation. This message indicates that such a conflict was detected, and denotes the attempts to resolve it.
Severity	4-Debug Level 4
Action	None.

Message	Assigned %n1 orphaned primary partitions
Parameters	%n1 - the number of orphaned primary partitions that were re-assigned
Severity	2-Warning
Cause	This service member (the most senior storage-enabled) has detected that one or more partitions have no primary owner (orphaned), most likely due to several nodes leaving the service simultaneously. The remaining service members agree on the partition ownership, after which the storage-senior assigns the orphaned partitions to itself. This message is followed by a list of the assigned orphan partitions. This message indicates that data in the corresponding partitions may have been lost.
Action	None.

Message	validatePolls: This service timed-out due to unanswered handshake request. Manual intervention is required to stop the members that have not responded to this Poll
Parameters	none
Severity	1-Error
Cause	When a node joins a clustered service, it performs a handshake with each clustered node running the service. A missing handshake response prevents this node from joining the service. Most commonly, it is caused by an unresponsive (e.g. deadlocked) service thread.
Action	Corrective action may require locating and shutting down the JVM running the unresponsive service. See Metalink Note 845363.1 for more details.

Message	java.lang.RuntimeException: Storage is not configured
Parameters	None
Severity	1-Error
Cause	A cache request was made on a service that has no storage-enabled service members. Only storage-enabled service members may process cache requests, so there must be at least one storage-enabled member.
Action	Check the configuration/deployment to ensure that members intended to store cache data are configured to be storage-enabled. This is controlled by the configuration element, or by the `-Dtangosol.coherence.distributed.localstorage` command-line override.

Message	An entry was inserted into the backing map for the partitioned cache "%s" that is not owned by this member; the entry will be removed."
Parameters	%s - the name of the cache into which insert was attempted
Severity	1-Error
Cause	The backing map for a partitioned cache may only contain keys that are owned by that member. Cache requests are routed to the service member owning the requested keys, ensuring that service members will only process requests for keys which they own. This message indicates that the backing map for a cache detected an insertion for a key which is not owned by the member. This is most likely caused by a direct use of the backing-map as opposed to the exposed cache APIs (e.g. ${xhtml}) in user code running on the cache server. This message is followed by a Java exception stack trace showing where the insertion was made.
Action	Examine the user-code implicated by the stack-trace to ensure that any backing-map operations are safe. This error can be indicative of an incorrect implementation of ${xhtml}

Message	Exception occured during filter evaluation: %s; removing the filter...
Parameters	%s - the description of the filter that failed during evaluation
Severity	1-Error
Cause	An exception was thrown while evaluating a filter for a ${xhtml} registered on this cache. As a result, some MapEvents may not have been issued. Additionally, to prevent further failures, the filter (and associated MapListener) will be removed. This message is followed by a Java exception stack trace showing where the failure occurred.
Action	Review filter implementation and the associated stack trace for errors.

Message	Exception occured during event transformation: %s; removing the filter...
Parameters	%s - the description of the filter that failed during event transformation
Severity	1-Error
Cause	An Exception was thrown while the specified filter was transforming a ${xhtml} for a ${xhtml} registered on this cache. As a result, some MapEvents may not have been issued. Additionally, to prevent further failures, the filter (and associated MapListener) will be removed. This message is followed by a Java exception stack trace showing where the failure occurred.
Action	Review filter implementation and the associated stack trace for errors.

Message	Exception occurred during index rebuild: %s
Parameters	%s - the stack trace for the exception that occurred during index rebuild*
Severity	1-Error
Cause	An Exception was thrown while adding or rebuilding an index. A likely cause of this is a faulty ${xhtml} implementation. As a result of the failure, the associated index is removed. This message is followed by a Java exception stack trace showing where the failure occurred.
Action	Review the ValueExtractor implementation and associated stack trace for errors.

Message	Exception occurred during index update: %s
Parameters	%s - the stack trace for the exception that occurred during index update
Severity	1-Error
Cause	An Exception was thrown while updating an index. A likely cause of this is a faulty ${xhtml} implementation. As a result of the failure, the associated index is removed. This message is followed by a Java exception stack trace showing where the failure occurred.
Action	Review the ValueExtractor implementation and associated stack trace for errors.

Message	Exception occurred during query processing: %s
Parameters	%s - the stack trace for the exception that occurred while processing a query
Severity	1-Error
Cause	An Exception was thrown while processing a query. A likely cause of this is an error in the implementation of the ${xhtml} used by the query. This message is followed by a Java exception stack trace showing where the failure occurred.
Action	Review the Filter implementation and associated stack trace for errors.

Message	BackingMapManager %s1: returned "null" for a cache: %s2
Parameters	%s1 - the classname of the ${xhtml} implementation that returned a null backing-map; %s2 - the name of the cache for which the BackingMapManager returned null
Severity	1-Error
Cause	A BackingMapManager returned `null` for a backing-map for the specified cache.
Action	Review the specified BackingMapManager implementation for errors and to ensure that it will properly instantiate a backing map for the specified cache.

Message	BackingMapManager %s1: failed to instantiate a cache: %s2
Parameters	%s1 - the classname of the ${xhtml} implementation that failed to create a backing-map; %s2 - the name of the cache for which the BackingMapManager failed
Severity	1-Error
Cause	A BackingMapManager unexpectedly threw an Exception while attempting to instantiate a backing-map for the specified cache.
Action	Review the specified BackingMapManager implementation for errors and to ensure that it will properly instantiate a backing map for the specified cache.

Message	BackingMapManager %s1: failed to release a cache: %s2
Parameters	%s1 - the classname of the ${xhtml} implementation that failed to release a backing-map; %s2 - the name of the cache for which the BackingMapManager failed
Severity	1-Error
Cause	A BackingMapManager unexpectedly threw an Exception while attempting to release a backing-map for the specified cache.
Action	Review the specified BackingMapManager implementation for errors and to ensure that it will properly release a backing map for the specified cache.

Message	Unexpected event during backing map operation: key=%s1; expected=%s2; actual=%s3
Parameters	%s1 - the key being modified by the cache; %s2 - the expected backing-map event from the cache operation in progress; %s3 - the actual MapEvent received
Severity	6-Debug Level 6
Cause	While performing a cache operation, an unexpected MapEvent was received on the backing-map. This indicates that a concurrent operation was performed directly on the backing-map and is most likely caused by direct manipulation of the backing-map as opposed to the exposed cache APIs (e.g. ${xhtml}) in user code running on the cache server.
Action	Examine any user-code that may directly modify the backing map to ensure that any backing-map operations are safe.

Message	Application code running on "%s1" service thread(s) should not call %s2 as this may result in deadlock. The most common case is a CacheFactory call from a custom CacheStore implementation.
Parameters	%s1 - the name of the service which has made a re-entrant call; %s2 - the name of the method on which a re-entrant call was made
Severity	2
Cause	While executing application code on the specified service, a re-entrant call (a request to the same service) was made. Coherence does not support re-entrant service calls, so any application code (CacheStore, EntryProcessor, etc.) running on the service thread(s) should avoid making cache requests. See the Constraints on Re-entrant Calls for more details.
Action	Remove re-entrant calls from application code running on the service thread(s) and consider using alternative design strategies as outlined in the Constraints on Re-entrant Calls

Message	Repeating %s1 for %n1 out of %n2 items due to re-distribution of %s2
Parameters	%s1 - the description of the request that must be repeated; %n1 - the number of items that are outstanding due to re-distribution; %n2 - the total number of items requested; %s2 - the list of partitions that are in the process of re-distribution and for which the request must be repeated
Severity	5-Debug Level 5
Cause	When a cache request is made, the request is sent to the service members owning the partitions to which the request refers. If one or more of the partitions that a request refers to is in the process of being transferred (e.g. due to re-distribution), the request is rejected by the (former) partition owner and is automatically resent to the new partition owner.
Action	None.

Message	Error while starting cluster: com.tangosol.net.RequestTimeoutException: Timeout during service start: ServiceInfo(%s)
Parameters	%s - information on the service that could not be started
Severity	1-Error
Cause	When joining a service, every service in the cluster must respond to the join request. If one or more nodes have a service that does not respond within the timeout period, the join times out.
Action	See Metalink Note 845363.1

Message	Failed to restart services: com.tangosol.net.RequestTimeoutException: Timeout during service start: ServiceInfo(%s)
Parameters	%s - information on the service that could not be started
Severity	1-Error
Cause	When joining a service, every service in the cluster must respond to the join request. If one or more nodes have a service that does not respond within the timeout period, the join times out.
Action	See Metalink Note 845363.

TCMP Log Messages

Message	Experienced a %n1 ms communication delay (probable remote GC) with Member %s
Parameters	%n1 - the latency in milliseconds of the communication delay; %s the full Member information
Severity	2-Warning or 5-Debug Level 5 or 6-Debug Level 6 depending on the length of the delay
Cause	This node detected a delay in receiving acknowledgment packets from the specified node, and has determined that is it likely due to a remote GC (rather than a local GC). This message indicates that the overdue acknowledgment has been received from the specified node, and that it has likely emerged from its GC.
Action	Prolonged and frequent GC's can adversely affect cluster performance and availability. If these warnings are seen frequently, review your JVM heap and GC configuration and tuning. See the performance tuning guide for more details.
Message	Failed to satisfy the variance: allowed=%n1 actual=%n2
Parameters	%n1 - the maximum allowed latency in milliseconds; %n2 - the actual latency in milliseconds
Severity	3-Informational or 5-Debug Level 5 depending on the message frequency
Cause	One of the first steps in the Coherence cluster discovery protocol is the calculation of the clock difference between the new and the senior nodes. This step assumes a relatively small latency for peer-to-peer round trip UDP communications between the nodes. By default, the configured maximum allowed latency (the value of the "maximum-time-variance" configuration element) is 16 milliseconds. Failure to satisfy that latency causes this message to be logged and increases the latency threshold, which will be reflected in a follow up message.
Action	If the latency consistently stays very high (over 100 milliseconds), consult your network administrator and run the Datagram Test.

Message	Created a new cluster "%s1" with Member(%s2)
Parameters	%s1 - the cluster name; %s2 - the full Member information
Severity	3-Informational
Cause	This Coherence node attempted to join an existing cluster the configured amount of time (specified by the "multicast-listener/join-timeout-milliseconds" element), but did not receive any responses from any other node. As a result, it created a new cluster with the specified name (either configured by the "member-identity/cluster-name" element or calculated based on the multicast listener address and port or the "well-known-address" list). The Member information includes the node id, creation timestamp, unicast address and port, location, process id, role, etc.)
Action	None, if this node is expected to be the first node in the cluster. Otherwise, the operational configuration has to be reviewed to determine the reason that this node does not join the existing cluster.
Message	This Member(%s1) joined cluster "%s2" with senior Member(%s3)
Parameters	%s1 - the full Member information for this node; %s2 - the cluster name; %s3 - the full Member information for the cluster senior node
Severity	3-Informational
Cause	This Coherence node has joined an existing cluster.
Action	None, if this node is expected to join an existing cluster. Otherwise, identify the running cluster and consider corrective actions.

Message	Member(%s) joined Cluster with senior member %n
Parameters	%s - the full Member information for a new node that joined the cluster this node belongs to; %n - the node id of the cluster senior node
Severity	5-Debug Level 5
Cause	A new node has joined an existing Coherence cluster.
Action	None.
Message	Member(%s) left Cluster with senior member %n
Parameters	%s - the full Member information for a node that left the cluster; %n - the node id of the cluster senior node
Severity	5-Debug Level 5
Cause	A node has left the cluster. This departure could be caused by the programmatic shutdown, process termination (normal or abnormal), or any other communication failure (e.g. a network disconnect or a very long GC pause). This message reports the node's departure.
Action	None, if the node departure was intentional. Otherwise, the departed node logs should be analyzed.

Message	MemberLeft notification for Member %n received from Member(%s)
Parameters	%n - the node id of the departed node; %s - the full Member information for a node that left the cluster
Severity	5-Debug Level 5
Cause	When a Coherence node terminates, this departure is detected by nodes earlier than others. Most commonly, a node connected via the TCP ring connection ("TCP ring buddy") would be the first to detect it. This message provides the information about the node that detected the departure first.
Action	None, if the node departure was intentional. Otherwise, the logs for both the departed and the detecting nodes should be analyzed.
Message	Service %s joined the cluster with senior service member %n
Parameters	%s - the service name; %n - the senior service member id
Severity	5-Debug Level 5
Cause	When a clustered service starts on a given node, Coherence initiates a handshake protocol between all cluster nodes running the specified service. This message serves as an indication that this protocol has been initiated. If the senior node is not known at this time, it will be shown as "n/a".
Action	None.

Message	This node appears to have partially lost the connectivity: it receives responses from MemberSet(%s1) which communicate with Member(%s2), but is not responding directly to this member; that could mean that either requests are not coming out or responses are not coming in; stopping cluster service.
Parameters	%s1 - set of members that can communicate with the member indicated in %s2; %s2 - member that can communicate with set of members indicated in %s1
Severity	1-Error
Cause	The communication link between this member and the member indicated by %s2 has been broken. However, the set of witnesses indicated by %s1 report no communication issues with %s2. It is therefore assumed that this node is in a state of partial failure, thus resulting in the shutdown of its cluster threads.
Action	Corrective action is not necessarily required, since the rest of the cluster presumably is continuing its operation and this node may recover and rejoin the cluster. On the other hand, it may warrant an investigation into root causes of the problem (especially if it is recurring with some frequency).
Message	validatePolls: This senior encountered an overdue poll, indicating a dead member, a significant network issue or an Operating System threading library bug (e.g. Linux NPTL): Poll
Parameters	none
Severity	2-Warning
Cause	When a node joins a cluster, it performs a handshake with each cluster node. A missing handshake response prevents this node from joining the service. The log message following this one will indicate the corrective action taken by this node.
Action	If this message reoccurs, further investigation into the root cause may be warranted.

Message	Received panic from senior Member(%s1) caused by Member(%s2)
Parameters	%s1 - the cluster senior member as known by this node; %s2 - a member claiming to be the senior member
Severity	1-Error
Cause	This occurs after a cluster is split into multiple cluster islands (usually due to a network link failure.) When a link is restored and the corresponding island seniors see each other, the panic protocol is initiated to resolve the conflict.
Action	If this issue occurs frequently, the root cause of the cluster split should be investigated.
Message	Member %n1 joined Service %s with senior member %n2
Parameters	%n1 - an id of the Coherence node that joins the service; %s - the service name; %n2 - the senior node for the service
Severity	5-Debug Level 5
Cause	When a clustered service starts on any cluster node, Coherence initiates a handshake protocol between all cluster nodes running the specified service. This message serves as an indication that the specified node has successfully completed the handshake and joined the service.
Action	None.
Message	Member %n1 left Service %s with senior member %n2
Parameters	%n1 - an id of the Coherence node that joins the service; %s - the service name; %n2 - the senior node for the service
Severity	5-Debug Level 5
Cause	When a clustered service terminates on some cluster node, all other nodes that run this service are notified about this event. This message serves as an indication that the specified clustered service at the specified node has terminated.
Action	None.

Message	Service %s: received ServiceConfigSync containing %n entries
Parameters	%s - the service name; %n - the number of entries in the service configuration map
Severity	5-Debug Level 5
Cause	As a part of the service handshake protocol between all cluster nodes running the specified service, the service senior member updates every new node with the full content of the service configuration map. For the partitioned cache services that map includes the full partition ownership catalog and internal ids for all existing caches. That same message is sent in the case of an abnormal service termination at the senior node, when a new node assumes the service seniority. This message serves as an indication that the specified node has received that configuration update.
Action	None.
Message	TcpRing: connecting to member %n using TcpSocket{%s}
Parameters	%s - the full information for the TcpSocket that serves as a TcpRing connector to another node; %n - the node id to which this node has connected
Severity	5-Debug Level 5
Cause	For quick process termination detection Coherence utilizes a feature called TcpRing, which is a sparse collection of TCP/IP-based connection between different nodes in the cluster. Each node in the cluster is connected to at least one other node, which (if at all possible) is running on a different physical box. This connection is not used for any data transfer; only trivial "heartbeat" communications are sent once a second per each link. This message indicates that the connection between this and specified node is initialized.
Action	None.

Message	Rejecting connection to member %n using TcpSocket{%s}
Parameters	%n - the node id that tries to connect to this node; %s - the full information for the TcpSocket that serves as a TcpRing connector to another node
Severity	4-Debug Level 4
Cause	Sometimes the TCP Ring daemons running on different nodes could attempt to join each other or the same node at the same time. In this case, the receiving node may determine that such a connection would be redundant and reject the incoming connection request. This message is logged by the rejecting node when this happens.
Action	None.
Message	Timeout while delivering a packet; requesting the departure confirmation for Member(%s1) by MemberSet(%s2)
Parameters	%s1 - the full Member information for a node that this node failed to communicate with; %s2 - the full information about the "witness" nodes that are asked to confirm the suspected member departure
Severity	2-Warning
Cause	Coherence uses UDP for all data communications (mostly peer-to-peer unicast), which by itself does not have any delivery guarantees. Those guarantees are built into the cluster management protocol used by Coherence (TCMP). The TCMP daemons are responsible for acknowledgment (ACK or NACK) of all incoming communications. If one or more packets are not acknowledged within the ACK interval ("ack-delay-milliseconds"), they are resent. This repeats until the packets are finally acknowledged or the timeout interval elapses ("timeout-milliseconds"). At this time, this message is logged and the "witness" protocol is engaged, asking other cluster nodes whether or not they experience similar communication delays with the non-responding node. The witness nodes are chosen based on their roles and location.
Action	Corrective action is not necessarily required, since the rest of the cluster presumably is continuing its operation and this node may recover and rejoin the cluster. On the other hand, it may warrant an investigation into root causes of the problem (especially if it is recurring with some frequency).

Message	This node appears to have become disconnected from the rest of the cluster containing %n nodes. All departure confirmation requests went unanswered. Stopping cluster service.
Parameters	%n - the number of other nodes in the cluster this node was a member of
Severity	1-Error
Cause	Sometime a node that lives within a valid Java process, stops communicating to other cluster nodes. (Possible reasons include: a) network failure; b) extremely long GC pause; c) swapped out process. ) In that case, other cluster nodes may choose to revoke the cluster membership fro the paused node and completely shun any further communication attempts by that node, causing this message be logged when the process attempts to resume cluster communications.
Action	Corrective action is not necessarily required, since the rest of the cluster presumably is continuing its operation and this node may recover and rejoin the cluster. On the other hand, it may warrant an investigation into root causes of the problem (especially if it is recurring with some frequency).
Message	A potential communication problem has been detected. A packet has failed to be delivered (or acknowledged) after %n1 seconds, although other packets were acknowledged by the same cluster member (Member(%s1)) to this member (Member(%s2)) as recently as %n2 seconds ago. Possible causes include network failure, poor thread scheduling (see FAQ if running on Windows), an extremely overloaded server, a server that is attempting to run its processes using swap space, and unreasonably lengthy GC times.
Parameters	%n1 - The number of seconds a packet has failed to be delivered or acknowledged; %s1 - the recipient of the packets indicated in the message; %s2 - the sender of the packets indicated in the message; %n2 - the number of seconds since a packet was delivered successfully between the two members indicated above
Severity	2-Warning
Cause	Possible causes are indicated in the text of the message.
Action	If this issue occurs frequently, the root cause should be investigated.

Message	Node %s1 is not allowed to create a new cluster; WKA list: [%s2]
Parameters	%s1 - Address of node attempting to join cluster; %s2 - List of WKA addresses
Severity	1-Error
Cause	The cluster is configured to use WKA, and there are no nodes present in the cluster that are in the WKA list.
Action	Ensure that at least one node in the WKA list exists in the cluster, or add this node's address to the WKA list.
Message	This member is configured with a compatible but different WKA list then the senior Member(%s). It is strongly recommended to use the same WKA list for all cluster members.
Parameters	%s - the senior node of the cluster
Severity	2-Warning
Cause	The WKA list on this node is different than the WKA list on the senior node.
Action	Ensure that every node in the cluster has the same WKA list.

Message	UnicastUdpSocket failed to set receive buffer size to %n1 packets (%n2 bytes); actual size is %n3 packets (%n4 bytes). Consult your OS documentation regarding increasing the maximum socket buffer size. Proceeding with the actual value may cause sub-optimal performance.
Parameters	%n1 - the number of packets that will fit in the buffer that Coherence attempted to allocate; %n2 - the size of the buffer Coherence attempted to allocate; %n3 - the number of packets that will fit in the actual allocated buffer size; %n4 - the actual size of the allocated buffer
Severity	2-Warning
Cause	See OS Performance Tuning
Action	See OS Performance Tuning

Coherence: POF Exceptions

2011-10-06T13:01:00.000-05:00

If you are trying to use POF for object serialization as recommended for better performance and less RAM utilization with Coherence, there are 2 ways to implement it:

1. Implement PortableObject interface for all objects that will be stored in the Coherence Grid - Need to modify the object class and implement readExternal and writeExternal methods.
2. Externalize the serialization logic by creating classes that implement PofSerializer interface and logic for serialization and de-serialization of your objects is implemented in this class using the serialize/deserialize methods. Later, you can map this class as the serializer for your object in pof-config.xml

More details on both the above ways can be found here.

If you are getting an error while using the PortableObject interface as below:

Caused by: java.io.EOFException: user type POF stream terminated at com.tangosol.io.pof.PofBufferWriter$UserTypeWriter.writeUserTypeInfo(PofBufferWriter.java:2621) at com.tangosol.io.pof.PofBufferWriter$UserTypeWriter.writeRemainder(PofBufferWriter.java:2497) at com.tangosol.io.pof.PortableObjectSerializer.serialize(PortableObjectSerializer.java:96) at com.tangosol.io.pof.PofBufferWriter.writeUserType(PofBufferWriter.java:1667)
... 13 more

Remember to remove the readRemainder() and writeRemainder() from the readExternal/writeExternal methods as they are used with POFSerializer interface.

Coherence Random Questions?

2011-08-28T17:14:00.000-05:00

What will happen if the eviction policy dictates that we evict an entry from the backing map that is still in the write-behind queue (and therefore has not been flushed to the database)?

In this situation, the read-write backing map will synchronously invoke the store operation on any entries about to be evicted. The implication here is that the client thread performing the put operation will be blocked while evicted entries are flushed to the database. It is an unfortunate side effect for the client thread, as its operation will experience a higher than expected latency, but it acts as a necessary throttle to avoid losing data. This edge condition highlights the necessity to configure a worker thread pool even for caches that are strictly performing write behind in order to prevent this flush from occurring on the service thread. It is important to keep in mind that the store operation will not always necessarily be performed by the write-behind thread. Note that this can also occur with caches that have expiry configured. The likelihood of this occurring will decrease if there is a large difference between expiry time and write-behind time.

What are service threads and how is the PUT operation managed in the Coherence Grid?

Each clustered service in Coherence is represented by a service thread at each JVM participating in the cluster. This thread is responsible for communicating with other nodes and providing the functionality exposed via the NamedCache API along with system level functionality such as life-cycle, distribution and fail-over. As a rule, all communications between the service threads are done in an asynchronous (non-blocking) mode, allowing for a minimum processing latency at this tier. On the other hand, the client functionality (for example a NamedCache.put call) is quite often implemented using a synchronous (blocking) approach using the internal poll API. Naturally, this poll API is not allowed to be used by the service thread, since this could lead to a high latency at best and deadlock at worst. When a listener is added to a local map that is used as a primary storage for the partitioned cache service, the events that such a listener receives are sent synchronously on that same service thread. It is discouraged to perform any operations that have a potential of blocking during such an event processing. A best practice is to queue the event and process it asynchronously on a different thread.

Coherence - Large clusters hanging in queries

2011-08-22T11:23:00.000-05:00

There will be instances where you might need to query the entire coherence cluster spanning 100+ nodes and you might experience that the queries are not completed in hours.

Filter filter =
Set entries = CacheFactory.getCache(cacheName).entrySet(filter);

The filters' should be constructed optimally and if the queries are spanning 100+ nodes then try using the PartitionedFilter for executing such queries. The Paritition filter limits the scope of another filter to those entries that have keys that belong to the specified partition set. This approach may complicate the client code, but can dramatically reduce the memory footprint used by the requestor.

Another approach to using PartitionedFilter is PartitionedIterator as below

PartitionedIterator iter = new PartitionedIterator(cache, filter, setPartitions,PartitionedIterator.OPT_ENTRIES | PartitionedIterator.OPT_BY_MEMBER);

while (iter.hasNext())
{
Map.Entry entry = (Map.Entry) iter.next();
}
}

What should be the value of thread-count in distributed cache?

2011-08-18T19:44:00.000-05:00

The thread-count value specifies the number of daemon threads used by the distributed cache service. If zero, all relevant tasks are performed on the service thread. The Default value is 0 and can be override using the system property tangosol.coherence.distributed.threads. This value will increase the parallelism of the processing in the Coherence Grid provided the edition used in Enterprise or Grid and not Standard.

It is recommended to set the value to 0 for scenarios with purely in-memory data (no read-through, write-through, or write-behind) and simple access (no entry processors, aggregators, and so on). For heavy compute scenarios (such as aggregators), the number of threads should be the number of available cores for that compute. For example, if you run 4 nodes on a 16 core box, then there should be roughly 4 threads in the pool. For IO intensive scenarios (such as read through, write-through, and write-behind), the number of threads must be higher. In this case, increase the threads just to the point that the box is saturated.

Remember, each service instance has its own primary thread. This thread has the option of using its own isolated thread pool if the thread-count is greater than zero. If the thread-count is zero, then all work will be performed by the primary service thread. If the thread-count is greater than zero, then all work will be performed by the thread pool (the primary thread acts as a task coordinator). The thread-count is per-service and per-cluster-member. Each cache service has a unique name. The CacheFactory class uses a single cache service instance for each cache type (Replicated/Distributed/etc). If you manually create additional cache services, they will each have their own isolated thread pools.

Oracle Coherence - Split Brain Scenario

2011-08-15T14:28:00.000-05:00

Please refer to Oracle Notes :

Oracle Coherence and Split-Brain FAQ [ID 1069132.1]

Oracle Coherence, Split-Brain, and Recovery Protocols Example In Detail [ID 1069429.1]

Witness Protocol: The Coherence clustering protocol (TCMP) is a reliable transport mechanism built on UDP. In order for the protocol to be reliable, it requires an acknowledgement (ACK) for each packet delivered. If a packet fails to be acknowledged within the configured timeout period, the Coherence cluster member will log a packet timeout. When this occurs, the cluster member will consult with other members to determine who is at fault for the communication failure. If the witness members agree that the suspect member is at fault, the suspect is removed. If the witnesses unanimously disagree, the accuser is removed. This process is known as the witness protocol.

Panic Protocol: When the presence of more than one cluster (i.e. Split-Brain) is detected by a Coherence member, the panic protocol is invoked in order to resolve the conflicting clusters and consolidate into a single cluster. The protocol consists of the removal of smaller clusters until there is one cluster remaining. In the case of equal size clusters, the one with the older Senior Member will survive.