OSB 11g: Stuck Threads when using inbound database adapter

Using a polling database adapter in a OSB proxy service, one may have noticed the following behaviour in Weblogic server:

• an exception in the server logs about one or even more stuck threads like this:

<BEA-000337> <[STUCK] ExecuteThread: ’10′ for queue: ‘weblogic.kernel.Default (self-tuning)’ has been busy for “705″ seconds working on the request “weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl@21b9db0″, which is more than the configured time (StuckThreadMaxTime) of “600″ seconds. Stack trace:
Thread-118 “[STUCK] ExecuteThread: ’10′ for queue: ‘weblogic.kernel.Default (self-tuning)’” <alive, suspended, waiting, priority=1, DAEMON> {
— Waiting for notification on: oracle.tip.adapter.db.InboundWork@21b8e86[fat lock]
java.lang.Object.wait(Object.java:???)
oracle.tip.adapter.db.InboundWork.run(InboundWork.java:498)
oracle.tip.adapter.db.inbound.InboundWorkWrapper.run(InboundWorkWrapper.java:43)
weblogic.work.ContextWrap.run(ContextWrap.java:39)
weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:528)
weblogic.work.ExecuteThread.execute(ExecuteThread.java:203)
weblogic.work.ExecuteThread.run(ExecuteThread.java:170)
}>

• and/or server health state from OSB managed server changing from state “Ok” to state “Warning”

Such a behaviour alerts administrators, thinking that something is wrong with the deployed applications or OSB services.

Looking in the Oracle documentation one can find the information that this is behaviour by design and that it can be ignored. To verify the OSB proxy service’s database adapter as the source for this the proxy service has  to be simply disabled in OSB console. Doing so makes the stuck threads disappear. The behaviour seems strange at the  - so why this?

When defining an inbound database adapter, Weblogic threads are used to perform the polling on events occurring in the defined database. Because OSB is designed to deliver high performance and throughput, a number of threads, which depends on the numberOfThreads property in the adapter’s JCA file, is exclusively reserved for the database adapter to perform the inbound polling. Such reserved threads will, due to performance reasons, never be released and never be returned to the corresponding thread pool. After the configured thread timeout, which is by default about 600 seconds, the stuck thread behaviour occurs.

Although this is the default behaviour, it is really confusing and could lead into serious problems, if real threading problems occur in deployed applications/services that are not caused by the adapter threads and which will not be noticed and handled in time. So what can be done to get rid of the stuck adapter threads?

The Oracle documentation proposes to define a separate Work manager and configure this with the proxy service transport’s dispatch policy. To do so, the following steps has to be performed:

• Define a custom global Work manager using weblogic console with the OSB managed server as deployment target

• Configure the new defined Work manager to ignore stuck threads

• Configure OSB proxy service transport’s dispatch policy to use the new defined Work manager

Afterwards the stuck threads behaviour caused the OSB proxy service or by its configured inbound database adapter should not show up again.

BEA-003108: Unicast receive error: java.io.EOFException

Technology, KeyWords:

WebLogic Server 10.3, Cluster, Unicast, Exception;JAVA.IO.EOFEXCEPTION

Error Message: <BEA-003108> <Unicast receive error : java.io.EOFException” >

Problem:

If I restart one Managed Server in a Cluster, I get the following error:

<Error> <Cluster> <BEA-003108> <Unicast receive error : java.io.EOFException

java.io.EOFException

at java.io.DataInputStream.readFully(DataInputStream.java:180)

at java.io.DataInputStream.readLong(DataInputStream.java:399)

at java.io.ObjectInputStream$BlockDataInputStream.readLong(ObjectInputStream.java:2799)

at java.io.ObjectInputStream.readLong(ObjectInputStream.java:960)

at weblogic.cluster.HeartbeatMessage.readExternal(HeartbeatMessage.java:55)

Truncated. see log file for complete stacktrace>

Description

Weblogic declaration from: http://docs.oracle.com/cd/E17904_01/apirefs.1111/e14397/ClusterExtension.html

<BEA-003108> <Unicast receive error : java.io.EOFException”

Error: Unicast receive error: e

An error occurred while trying to receive a message over the cluster broadcast.

Cause

An error occurred while trying to receive a message over the cluster broadcast.

Action

Make sure that the NIC is functioning properly. If you believe there no environment problems exist, contact Oracle Customer Support and provide the stack trace for further analysis.

More detail and Background information:

Note: Sometimes you get Error Message: <BEA-002616> <Failed to listen on channel “Default”…> and it can be a logical effect of error BEA-003108.

The issue ” <BEA-003108> <Unicast receive error : java.io.EOFException”  occurs when using a Unicast cluster and when the cluster debug flags DebugCluster and/or DebugClusterHeartBeats are turned on for some of the managed servers, but not all of them.

The issue also occurs when the debug flags are turned on for the servers that existed previously in the cluster but not on servers which are newly added.

Solution

To resolve this issue, the cluster debug flags need to be enabled consistently: either disabled on all the servers of the cluster or enabled on all of them. In a production environment, debugging should be disabled. Cluster debug flags should be enabled only while debugging a problem with the Unicast cluster.

Solution via AdminServer:

1. Click on Domain Structure and select your Server
2. Select debugging tab
3. Select items, that you want to disable or/enable, e.g. Disable weblogic debugging by click on „disable“(german: „deaktivieren“)
4. Check Log files, wether the issue is solved.

Refernces

• Screenshots: http://modjorg.wordpress.com/2013/01/28/bea-003108-unicast-receive-error-java-io-eofexception/
• http://docs.oracle.com/cd/E17904_01/apirefs.1111/e14397/ClusterExtension.html
• http://modj.org/home/informatik/middleware/weblogic.html
• MOS: WebLogic Server: Getting”Unicast receive error: java.io.EOFException” Exceptions in Logs [ID 1289389.1]

BEA-002616: Failed to listen on channel on listenAddress:port

Autor: Mohammad Esad-Djou

Technology, KeyWords:

WebLogic Server 10.3, Cluster, Exception; Too many open files, Socket, Linux, SLES, Error Message: <BEA-002616> <Failed to listen on channel “Default”…>

Problem:

<Critical> <Server> <BEA-002616> <Failed to listen on channel “Default” on IP_NR:Port, failure count: 1, failing for 0 seconds, java.net.SocketException: Too many open files>

Description

Weblogic declaration from: http://docs.oracle.com/cd/E23549_01/apirefs.1111/e14397/Server.html

BEA-002616

Critical: Failed to listen on channel “channel” on listenAddress:port, failure count: fails1, failing for secs2 seconds, e3

• Description: The server listener will retry the listen after a short delay.
• Cause: The server got an exception while trying to accept client connections. It will try to backoff to aid recovery.
• Action: The OS limit for the number of open file descriptor (FD limit) needs to be increased. Tune OS parameters that might help the server to accept more client connections (e.g. TCP accept back log).

More detail and Background information:

“Tune OS parameters” depends on OS which you use. For Linux, kernel parameters need adjustment- exact details depend on the distribution. There are individual user limits for open files. You can use `ulimit -a` command to find out the Linux for the user that owns.

You can find information regarding the important parameters e.g. number of “open files” or similar entry. Please consider, in AIX, the user limits apply as well as a system OS configuration for the total number of open files allowed on the host.

Solution

Here is an example how can you check and isolate the position of error.

1-    Check linux configuration (e.g. on host XXX):

> ulimit –a

core file size          (blocks, -c) 1

data seg size           (kbytes, -d) unlimited

scheduling priority             (-e) 0

file size               (blocks, -f) unlimited

pending signals                 (-i) 127575

max locked memory       (kbytes, -l) 64

max memory size         (kbytes, -m) 13887980

open files                      (-n) 65600 à OK

pipe size            (512 bytes, -p) 8

POSIX message queues     (bytes, -q) 819200

real-time priority              (-r) 0

stack size              (kbytes, -s) 8192

cpu time               (seconds, -t) unlimited

max user processes              (-u) 127575

virtual memory          (kbytes, -v) 17749200

file locks                      (-x) unlimited

The open file descriptor limit is at 65600 as recommended by Oracle.

2-    Check currently open files

• Find PID of Managed Server

> ps -fea|grep myManagedServer

• Check open files

Please use lsof (list open files) that lists information about files opened by processes

You see list of open files via (e.g.PID is here 1234)

>lsof -p 1234

In order to findour the number of open files:

> lsof -p 1234 | wc –l

In this case, we observed that application has 13 thousand connections on “CLOSE_WAIT” status. This usually happens when the application doesn’t close the connections properly:

CLOSE_WAIT

The remote end has shut down, waiting for the socket to close.

It’s reason why they are reaching the 65600 limit.

***

After more analyses this case, it is cleared, the issue is a result of weblogic cluster configuration.

“Open files” are waiting to close cluster communication between two nodes. See <BEA-003108> <Unicast receive error: java.io.EOFException” >

Refernces

• Additional information regarding <BEA-002616> is available on MOS: WebLogic Server Support Pattern: How To Troubleshoot Too Many Open Files Problems [ID 867492.1]
• http://docs.oracle.com/cd/E23549_01/apirefs.1111/e14397/Server.html
• http://modj.org/home/informatik/middleware/weblogic.html

Composite Management: Undeployment with WLST or ANT

Oracle BPM / SOA Suite offers different options to deploy or delete composites. You can use the Enterprise Manager, the Weblogic Scripting Tool (WLST) or ANT. In complex environments with many different composites and versions it is hard to undeploy them one by one and manually. If you are interested in a script solution for this visit my post under Inside BPM and SOA.

Weblogic Application Server – Free for developers

That’s good news and solves part of the licensing problem we always had: You don’t have to have a separate license of WLS for every developer, using it on their development machines. Find details in Bruno’s blog: https://blogs.oracle.com/brunoborges/entry/weblogic_server_free_for_developers

Java WebService-Client: Endpunkt-URL dynamisch setzen basierend auf der Adresse des WLS Managed Servers

Während man die Zieladressen von Service-Referenzen innerhalb eines Composites komfortabel per Config-Plan ändern kann, funktioniert dieses Vorgehen nicht für Java Web-Service Clients – selbst wenn diese Teil eines Composites sind (z.B. innerhalb eines Mediator Callouts).

Oft hat man mehrere Umgebungen, z.B. Test, Integration und Produktion, und es soll jeweils so konfiguriert werden, dass die Services auf demselben Host aufgerufen werden. Der offensichtliche Weg wäre, als Adresse einfach “localhost” zu verwenden. Das bringt allerdings ein paar Probleme mit sich, etwa dass man dann aus der Entwicklungsumgebung heraus nicht testen kann oder, dass es nicht funktioniert, weil der Weblogic gar nicht auf dem Loopback-Interface “lauscht”.

Eine Möglichkeit zur Lösung des Problems ist, jeweils dynamisch die Endpunkt-Adresse zu setzen, basierend auf der im Weblogic hinterlegten Listen Address. Dies passiert, nachdem man den Port des Webservices geholt hat, jedoch bevor man den Call auf dem Port ausführt. Die “DefaultURL” des lokalen WLS bekommt man aus einem MBean. Sobald man diese hat, setzt man auf dem Port die entsprechende Property für die Endpoint Adresse.

Ein Beispiel für eine entsprechende Java-Methode:

private void setEndpointAddressToLocal(MeinService_ptt port) throws Exception {
 // Obtain local listen address
 ObjectName service = new ObjectName(
  "com.bea:Name=RuntimeService,Type=weblogic.management.mbeanservers.runtime.RuntimeServiceMBean");
                                                                                     
 InitialContext ctx = new InitialContext();
 MBeanServer mBeanServer = (MBeanServer) ctx.lookup("java:comp/env/jmx/runtime");

 ObjectName rt = (ObjectName) mBeanServer.getAttribute(service, "ServerRuntime");
 String defaultUrl = (String) mBeanServer.getAttribute(rt, "DefaultURL");
 String newAddress = defaultUrl.replaceAll("t3://", "http://");       
 newAddress += "/soa-infra/services/default/MeinServiceComposite/MeinService";
       
 log.info("Setting endpoint address to " + newAddress);
       
 // Set new address
 BindingProvider bp = (BindingProvider)port;
 Map context = bp.getRequestContext();
 context.put(BindingProvider.ENDPOINT_ADDRESS_PROPERTY, newAddress);
}

Vielen Dank an folgende Quellen, die bei der Lösungssuche sehr hilfreich waren:

• http://kingsfleet.blogspot.com/2007/11/changing-endpoint-location-for-jax-ws.html
• http://stackoverflow.com/questions/8208940/getting-instance-ip-from-weblogic-cluster
• Weblogic MBean-Browser im Enterprise Manager

WLS jdbc Anbindung von iSeries AS400/DB2

Allgemeines

In diesem Artikel wird die Verbindung zwischen einem WLS und einer DB2 Instanz , die in einem mainframe läuft.  Für eine DB2 Instanz, die unter linux, solaris etc. läuft gelten andere Regeln.

JDBC Treiber

In der iSeries As400 (Mainframe) gibt es das IFS Verzeichnis mit dem File:

/QIBM/ProdData/HTTP/Public/jt400/lib/jt400.jar

 Die Datei jt400.jar von dort downloaden, weil genau diese jdbc Datei ist mit dieser Version des BS kompatibel.
SOA Datasource erstellen:
jt400.jar unter

• user_projects/domains/DOMAINNAME/lib 

und

• <Weblogic_Home>/server/lib

kopieren.

Unter

<Weblogic_HOME>/common/bin/commEnv.sh

den classpath erweitern so, dass  jt400.jar berücksichtigt wird.

NonXA Datasource erstellen.

Tab Connection Pool:

URL	jdbc:as400://IP/library
Driver Class Name	com.ibm.as400.access.AS400JDBCDriver

Tab Transactions:
Supports Global Transactions ist deaktiviert

DB Adapter update

Danach wie gewöhnt den DB Adapter erweitern um den connection pool und updaten.

Hiermit dürfte die jdbc Verbindung zwischen SOA Suite und AS400/DB2 funktionieren.

Road to 12c – Oracle SOA Suite 12c und BPM Suite 12c

Die Reise der Oracle Fusion Middleware in Richtung 12c ist in vollem Gange.

Mit der Weblogic Application Server 12c ist die Basis der Fusion Middleware Familie bereits in der Zukunft angekommen. Die Featureliste des WLS 12c ist beeindruckend, die ersten praktischen Erfahrungen ebenso. Man merkt deutlich, dass die Integration des alten Oracle Application Servers, der BEA Containers und auch der Glassfish-Features nun zu einem sehr runden Produkt geführt hat. Es hat gedauert – aber das Warten hat sich gelohnt.

Wie aber sieht es bei der SOA Suite aus?

Diese wird vermutlich erst Anfang 2013 als 12c Release zur Verfügung stehen. Das Feature-Set ist hochbeeindruckend. Die Basisfunktionalitäten sind in 11g bereits komplett, in 12c kommt all das hinzu, was das Leben deutlich einfacher macht. Unter dem Schlagwort „Industrial SOA” kommen Optimierungen in den Bereichen Performance&Scalability (Memory-Bedarf runter!), Diagnosability, Upgrade, Build-in Governance und Developer Productivity (Download&Installation mit One-Click-HelloWorld-Erfahrung).

Sollte man dann nun warten auf SOA Suite 12c und was passiert mit Migrationen von 11g auf 12c? Hier liegt die gute Nachricht. Oracle hat verstanden, was den Kunden in der Vergangenheit aufgestoßen ist. Es wird ein Seamless-Upgrade auf 12c einer laufenden 11g SOA Suite Plattform geben, inklusive Übernahme aller laufenden Instanzen. Super!

Und BPM Suite 12c?

Hier werden erste öffentliche Informationen in der Keynote auf den SOA, BPM, Integration Days direkt vom Oracle Product Management zu hören sein (http://soa-bpm-days.de/). Mit dem Patchset PS5 hat die BPM Suite 11g bereits nicht nur ein Fehlerupdate, sondern ein komplettes Feature-Update bekommen. In 12c, vermutlich zeitglich zur SOA Suite 12c, erwarten uns Dinge wie ACM, Social, erweiterte Modellierungswerkzeuge, optimiertes Modellieren direkt im Browser, uvm.

Fazit: Treffen Sie heute ihre Vorbereitungen, schaffen sie Mehrwerte über die 11g Plattform und freuen sie sich auf den Innovations-Boost, den die 12c Plattform ermöglichen wird. Und freuen sie sich auf das erste Major-Update, dass sie einfach so werden einspielen können. Es bleibt spannend…