Archiv

Archiv für die Kategorie ‘Weblogic Server’

IT-Security (Part 3): WebLogic Server and Java Security Features

WebLogic Server and Java Security Features [1]

WebLogic Server supports the Java SE and Java EE Security to protect the resources of whole system. The resources could be Web applications, Uniform Resource Locator (URL), Enterprise JavaBeans (EJBs), and Connector components.

Java SE capabilities: Security APIs

Java uses APIs to access security features and functionality and its architecture contains a large set of application programming interfaces (APIs), tools, and implementations of commonly-used security algorithms, and protocols. This delivers the developer a complete security framework for writing applications and enables them to extend the platform with new security mechanisms.[2]

Java Authentication and Authorization Services (JAAS)

WebLogic Server uses the Java Authentication and Authorization Service (JAAS) classes to consistently and securely authenticate to the client. JAAS is a part of Java SE Security APIs and a set of Java packages that enable services to authenticate and enforce access controls upon users and /or fat-client authentication for applications, applets, Enterprise JavaBeans (EJB), or servlets.

JAAS uses a Pluggable Authentication Module (PAM) framework, and permits the use of new or updated authentication technologies without requiring modifications to the application. Therefore, only developers of custom Authentication providers and developers of remote fat client applications need to be involved with JAAS directly. Users of thin clients or developers of within-container fat client applications do not require the direct use or knowledge of JAAS.

JAAS LoginModules

All LoginModules are responsible for authenticating users within the security realm (we are going to discuss about that later) and for populating a subject with the necessary principals (users/groups). LoginModules contains necessary methods for Login Context, Accounts, Credentials, configuration of them, and different ways to exception handling. Each Authentication providers will be configured in a security realm, its LoginModules will store principals within the same subject too. I try to present that with an example: Via WebLogic Server Admin Console: Home >myDomain > Domain Structure click on Security Realms and then create a new realm “Moh_Realm-0” and then click on “OK”

p3_realm_1

Figure 1 create a new Realm

Select new realm and then click on tab “provider”, and then click on “New”, in order to create a new provider:

p3_realm_2

Figure 2 open the new Realm

In this use case, we select type: “WebLogic Authentication Provider” and give a name e.g. “DefAuthN”, then “OK”.  The WebLogic Authentication provider is configured in the default security realm (myrealm). The WebLogic Authentication provider allows you to edit, list, and manage users, groups, and group membership. User and group information is stored in the embedded LDAP server.[3]

p3_AuthenticationProvider_3

 Figure 3 create a new Authentication Provider

After define “Provider”, we have to restart Admin Server. Now, we can check and compare users of new realm (Moh_Realm-0) with default realm (myrealm) of WebLogic. For myrealm, Icreated a new user named “userDOAG” and we see the following list there (Home >Summary of Security Realms >myrealm >Users and Groups)

p3_users_4

Figure 4 users of myrealm

But I didn’t create same user for Moh_Realm-0 (Home >DefAuthN>Summary of Security Realms >Moh_Realm-0 >Users and Groups):

p3_users_5

Figure 5 users of Moh_Realm-0

It shows, that we can use security provider in different gatherings und expand our security realm with additional user, groups, and security providers. We are working on it in next part of this article.

JAAS Control Flags

The JAAS Control Flag attribute determines how the LoginModule for the WebLogic Authentication provider is used in the login sequence. The values for the Control Flag attribute are as follows: Home >Summary of Security Realms > Moh_Realm-0 >Providers > DefAuthN

 p3_JAAS_ControlFlag_6

Figure 6 Control flags via Admin Consol

  • REQUIRED – This LoginModule must succeed. Even if it fails, authentication proceeds down the list of LoginModules for the configured Authentication providers. This setting is the default.
  • REQUISITE – This LoginModule must succeed. If other Authentication providers are configured and this LoginModule succeeds, authentication proceeds down the list of LoginModules. Otherwise, return control to the application.
  • SUFFICIENT – This LoginModule needs not succeed. If it does succeed, return control to the application. If it fails and other Authentication providers are configured, authentication proceeds down the LoginModule list
  • OPTIONAL – The user is allowed to pass or fail the authentication test of these Authentication providers. However, if all Authentication providers configured in a security realm have the JAAS Control Flag set to OPTIONAL, the user must pass the authentication test of one of the configured providers.[4]

Now, we can focus on two important JAAS-tasks: authentication and authorization of users…[5]

References


[4] Oracle Fusion Middleware: Understanding Security for Oracle WebLogic Server 12c Release 1, (12.1.1), E24484-02, January 2012: http://docs.oracle.com/cd/E24329_01/web.1211/e24484.pdf

IT-Security (Part 2): WebLogic Server and Oracle Platform Security Services (OPSS)

OPSS Architecture

As we discussed (http://modj.org/home/aktueles/it-security-weblogic-server-and-oracle-platform-security-services-opss/e17330b741d0e387ead1a36591466a7c.html), OPSS is Oracle proposals regarding enterprise security services. It is as a framework that provides a comprehensive set of security services. These services based on Java technologies and have a consistent approach for design and apply security policies to Java EE and resources. We look at OPSS architecture from two different perspectives, which are connected to each other very closely. I try to review the advantages of OPSS for developers and administrators from Application’s perspective and present the cooperating of technology components such as LDAP, Application Server and Oracle Fusion Middleware from Component’s perspective. Thereby, we can determine the main OPSS’s benefits that Oracle says:

  • Allows developers to focus on application and domain problems
  • Supports enterprise deployments
  • Supports several LDAP servers and SSO systems
  • Is certified on the Oracle WebLogic Server
  • Pre-integrates with Oracle products and technologies

Application’s point of view

Oracle Platform Security Services (OPSS) is both a security framework exposing security services and APIs, and a platform offering concrete implementation of security services. It includes these elements:

  • Common Security Services (CSS), the internal security framework on which Oracle WebLogic Server is based
  • Oracle Platform Services
  • User and Role APIs
  • Oracle Fusion Middleware Audit Framework

Figure 1 Application’s perspective  illustrations OPSS‘s architecture from application point of view. Such architecture allows OPSS to support different security and identity systems without changing the APIs. OPSS is integrated with Oracle Fusion Middleware‘s management tools to administrate and monitor the security policies implemented in the underlying identity management infrastructure.  Therefore, OFM technologies such as Oracle SOA, Oracle WebCenter Suite, Oracle Application Development Framework (ADF), Oracle Web Services Manager (OWSM) and… could use OPSS capacities.

OPSS offers abstraction layer APIs those isolate developers from security and identity management implementation details. In this way, developer can invoke the services provided by OPSS directly from the development environment (e.g. JDeveloper) using wizards. Admin can configure the services of OPSS into the WLS. As you see in Figure, the uppermost layer consists of Oracle WebLogic Server and the components and Java applications running on the server; below this is the API layer consisting of Authentication, Authorization, CSF (Credential Store Framework), and User and Role APIs, followed by the Service Provider Interface (SPI) layer and the service providers for authentication, authorization, and others. The final and bottom layer consists of repositories including LDAP and database servers.

Figure 1 Application's perspective

Figure 1 Application’s perspective

 OFM-Component’s point of view

Figure 2 OFM-Component’s perspective shows the various security components as layers. The top layer includes the OPSS security services; the next layer includes the service providers, and the bottom layer includes the OPSS security store with a repository of one of three kinds. OPSS provides auditing capabilities for components too.

The second layer [Security Services Provider Interface (SSPI)] has the capability that works with Java EE container security – named Java Authorization Contract for Containers (JACC) mode and in resource-based (non-JACC) mode, and resource-based authorization for the environment.

SSPI is a set of APIs for implementing pluggable security providers. A module implementing any of these interfaces can be plugged into SSPI to provide a particular type of security service. Therefore, OPSS has a consistent structure and is able to meet the requirements for integrating JEE Applications generally and specially OFM-Components and Oracle Security technologies, such as OAM, OID and so on.

Figure 2 OFM-Component's perspective

Figure 2 OFM-Component’s perspective

References

IT-Security: WebLogic Server and Oracle Platform Security Services (OPSS)

IT security is popular in a way never known before! I love it!

If I discussed e.g. in a WebLogic Server workshop about that, I heard normally form administrators: That’s not my thing, forget it! But newly, everybody wants to know “how can we secure our data and our information?!”  To be honest, you need to detect your application server that you are using, and if you are not able to use WebLogic Server security features, then this could be your problem.

WebLogic Server uses a security architecture that provides a unique and secure foundation for applications that are available via the Web. It is designed for a flexible security infrastructure and enabled to response the security challenges on the Intra- and Internet. We are able to use security capacity of WebLogic Server as a standalone feature to secure WebLogic Server and/or as part of a corporation-wide, security management system.

Overview

In order to achieve a satisfactory level of security, we have to design an integrated security policy: from lack of resources till the increasing complexity of IT systems. The elementary principles in IT security are Confidentiality and/or privacy, availability and integrity. Confidentiality and/or privacy mean information that has to be protected against unauthorized disclosure. Availability means services; IT system functions and information must be available to users when they need it. Integrity means data must be complete and unaltered.  Therefore, we understand security policy as a policy that it covers protection objectives and broad-spectrum security measures in the sense of the acknowledged requirements of an organization.

Simple to say, security is the protection of information that needs to protected, from unauthorized access. IT security could be helped us through technology, processes, policies and training, so that we can be sure that data stored and secured in a computer or passed between computers is not compromised.  Therefor data encryption is the first step in the direction IT-Security. In order to access to specific resources, user needs to provide (normally) his user name and password. Data encryption is the transformation of data into a form that cannot be understood without decryption key(s).

Security Challenges

In a world that we used to work with distributed IT-landscape, we face to with different challenges, e.g. network-based Attacks, heterogeneity on application layer from user interface till to application.  It is really difficult to stay on a standard security level for all of team members of development team. We cannot awaiting all of application developers to be able develop solve the security challenges such as privacy, identity management, compliance, audit too.  Another area is interfaces between application server and backend database.

A simple case is presented on the following diagram: most applications are multi-tiered and distributed over several systems. A client invokes an application or sends a request to server. This case presents how many systems are in transaction involve.  We have to check all of critical points and interfaces: network-based attacks, user interface, application Server and so on.

See: http://modjorg.files.wordpress.com/2014/02/security_challenges_1.jpg

On these grounds, we need to use an enterprise security framework that allows application developers to pick and choose from a full set of reusable and standards based security services that allow security, privacy, and audit. Oracle Platform Security Services (OPSS) is a security framework that runs on WebLogic Server and is available as part of WebLogic Server. It combines the security features of BEA‘s internal security (WLS + Oracle Entitlement Server (OES)) and the OAS (Hava Platform Security (JPS) – earlier JAZN) to provide application developers, system integrators, security administrators, and independent SW vendors with a comprehensive security platform framework for Java SE and Java EE applications. In this form, Oracle is able to suggest a uniform enterprise security policy and a self-contained and independent framework with Identity management and audit services across the enterprise. The heart of whole system beats on WebLogic Server.

WebLogic Server provides authentication, authorization, and encryption services with which you can guard these resources. These services cannot provide protection, however, from an intruder who gains access by discovering and exploiting a weakness in your deployment environment. Therefore, whether you deploy WebLogic Server on the Internet or on an intranet, it is a good idea to contact an independent security expert to go over your security plan and procedures, audit your installed systems, and recommend improvements.

References

Did you forget your WebLogic Server password? No problem!

It is a frequently question that I receive from time to time: “I’ve lost my password, how can I change the AdminServer Password in WebLogic Server?”  In addition, normally administrators want to change the WLS password regularly, because of the security policy. I try to make it easy, but please try it first in a test environment! Make a Backup from your domain and do the following steps for WebLogic Server 11g and/or 12c:

1-       Check your Middleware-Home’s ($MW_HOME) and WLS-Domain-Home’s ($DOMAIN_HOME) paths. You can set up the environments under Linux with export command, e.g.:

$>export MW_HOME=/u01/app/oracle/middleware

$>export DOMAIN_HOME=$MW_HOME/user_projects/domains/MyDomain

2-      shut down the WebLogic domain, e.g. $DOMAIN_HOME/bin/stopWebLogic.sh

3-      Rename the data folder in $DOMAIN_HOME/servers/AdminServer/data

4-      set the environment variables with using, e.g.: $DOMAIN_HOME/bin/setDomainEnv.sh

### Now is the main command that you can change AdminServer’s password:

$> cd $DOMAIN_HOME/security

$> java weblogic.security.utils.AdminAccount <username> <NEWpassword> .

# Create and/or update “boot.properties” in “$DOMAIN_HOME/servers/AdminServer/security/ directory with your username and NEWpassword:

username=<username>

password=<NEWpassword>

# Start the WebLogic domain.

$ $DOMAIN_HOME/bin/startWebLogic.sh

Refernces:

Kategorien:Weblogic Server Schlagworte:

PS4: Error for Updating OSB Domain after applying the latest Patch Set

Problem:

The customer does the following steps as PS4 Doc regarding “Updating OSB Domain after applying the latest Patch Set” (See 3.8.9.1 Updating an Oracle Service Bus Domain After Applying the Latest Patch Set in http://docs.oracle.com/cd/E21764_01/doc.1111/e16793/patch_set_installer.htm)

Perform the following steps for each domain to be upgraded:

1. Make sure you have backed up and shut down all domains to be upgraded.

2. Under each Oracle Service Bus 11gR1 domain to be upgraded, open a command window and run the DOMAIN/bin/setDomainEnv.cmd/sh command.

4. In the command window, switch to the directory in which the upgrade scripts resides: OSB_ORACLE_HOME/common/lib/upgrade

5. On the command line, runs the appropriate script for your operating system: Linux/Solaris: java weblogic.WLST ./domainUpgrade.py

java weblogic.WLST ./domainUpgrade.py

Initializing WebLogic Scripting Tool (WLST) …

Welcome to WebLogic Server Administration Scripting Shell

Type help() for help on available commands

AND he gets the following error:

Problem invoking WLST – Traceback (innermost last):

File “/oracle/fmw/Oracle_OSB1/common/lib/upgrade/./domainUpgrade.py”, line 368, in ?

File “/oracle/fmw/Oracle_OSB1/common/lib/upgrade/./domainUpgrade.py”, line 16, in replaceOSBServerModuleVersion

ValueError: substring not found in string.index

Solution:

This issue happens if the upgrade script has been run more than once and cannot find the proper substring in the configuration because it has already been changed.

„To resolve this issue first verify that the upgrade was not already run by examining the time-stamp of the DOMAIN_HOME/config/config.xml of the domain begin upgraded. If this file has recently changed then it is likely that the upgrade ran to completion. You can also look at the config.xml file and verify the version in the section:

<domain-version>10.3.4.0</domain-version>“

if you upgraded more than once, then you can solve the issue in two ways:

  1. Do not do anything and continue with the next steps of your upgrade, because script had already been run.
  2. (my suggestion) Please restore from backup and re-run your upgrade again, because you can be sure your upgrade is fully correct and you do not lost your time for problem analysis, if you find that there are other problems after the upgrade.

Refernces:

The following note will help you: Running The Domainupgrade.Py Script Gives Error: “substring not found in string.index” (Doc ID 1313321.1)

Using Credential Store Framework when communicating with Oracle Human Workflow API

17. Dezember 2013 1 Kommentar

For connecting to Oracle Human Workflow Engine via the provided client API, username and password of an admin user are needed. These credentials could also be useful during task processing, when actions on a task has to be performed on behalf of a user, for example in case of holidays or illness. But how can to manage the admin users credentials in secure way, independent from the target environment?

A first approach is to use a mechanism where the credentials were provided as context parameters in the web.xml, of a Facade Web Service in front of the client API to hide complexity and to force upgrade protection in case of API changes. When deploying this Web Service facade, the parameters are replaced using a deployment plan. This solution works, but has the disadvantage that username and password of the admin user are contained in the deployment plan as clear text. From a SysOps perspective this mechanism is not appropriate. 

So another possibility must be found to manage user credentials in a consistent and secure way. An approach to ensure the secure management of credentials is to use the Oracle Credential Store Framework (CSF), provided by Oracle Platform Security Services (OPSS). Configuring and using CSF is quite simple and done in a few steps:

1. Create Credentials Store in EM (Right click on Weblogic domain > [Domainname] and then choose Security > Credentials from the Dropdown menu)

Create_CS

2. Configure System Policy to authorize access to the configured Credential Store (Right click on Weblogic domain > [Domainname] and then choose Security > Credentials from the Dropdown menu)

Create_System_Policies

The configurations, needed to allow read-only access from an application, contains the following information

Type: Codebase
Codebase: file:${domain.home}/servers/${weblogic.Name}/tmp/_WL_user/<APPLICATION_NAME>/-
Permission:
  Permission Class: oracle.security.jps.service.credstore.CredentialAccessPermission
  Resource Name: context=SYSTEM,mapName=WORKLIST-API,keyName=*
  Permission Actions: read

3. Deploy the application

Managing the credentials in the Credential Store may also be done by using WLST functionalities, which would be more maintainable from a SysOps perspective. Details on that could be found here. The system policies may be directly edited in <MW_HOME>/user_projects/domains/<DOMAIN_NAME>/config/fmwconfig/system-jazn-data.xml. But this approach may be error-prone and often not appropriate in clustered production environments, when OPSS configuration is done in a database or LDAP.

Accessing the so configured Credential Store by a server application is done by using the lines of code below. For developing the shown CSF access, jps-api.jar must be in the classpath of the application. At runtime the needed dependencies are provided by Oracle Weblogic Server.

package com.opitzconsulting.bpm.connection;
import java.security.AccessController;
import java.security.PrivilegedActionException;
import java.security.PrivilegedExceptionAction;

import oracle.security.jps.service.JpsServiceLocator;
import oracle.security.jps.service.credstore.CredentialStore;
import oracle.security.jps.service.credstore.PasswordCredential;

public final class CsfAccessor {

 static PasswordCredential readCredentialsfromCsf(String pCsfMapName, String pCsfKey) {

   try {
     return AccessController.doPrivileged(new PrivilegedExceptionAction<PasswordCredential>() {

            @Override
            public PasswordCredential run() throws Exception {

              final CredentialStore credentialStore = JpsServiceLocator.getServiceLocator().lookup(CredentialStore.class)
              return (PasswordCredential) credentialStore.getCredential(pMapName, pKey);
            }
          });
   } catch (Exception e) {
     throw new RuntimeException(String.format("Error while retrieving information from credential store for Map [%s] and Key [%s]", pCsfMapName, pCsfKey), e);
   }
  }
}

When having more applications that need to access credentials from the Credentials Store, it is recommended to implement the access to CSF centrally and provide the functionality as a shared library within Weblogic Server. Otherwise you have to configure the corresponding System Policies, which authorizes the access to CSF, separate for every new application that needs to have access to CSF. Using the shared library approach, only the shared library itself has to be authorized for accessing the Credentials Store. Applications that need to access CSF must only specify the dependency to the shared library in the application’s deployment descriptor file, like weblogic-application.xml.

<wls:weblogic-application
  xmlns:wls="http://xmlns.oracle.com/weblogic/weblogic-application"
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://java.sun.com/xml/ns/javaee http://java.sun.com/xml/ns/javaee/javaee_5.xsd http://xmlns.oracle.com/weblogic/weblogic-application http://xmlns.oracle.com/weblogic/web  logic-application/1.3/weblogic-application.xsd">
  <wls:library-ref>
    <wls:library-name>csf-accessor-shared-lib</wls:library-name>
    <wls:implementation-version>1.0.0</wls:implementation-version>
  </wls:library-ref>
</wls:weblogic-application>

In order to encapsulate the access to CSF and to avoid the publication of the PasswordCredential object instance, we decided to further encapsulate the CSF access by a special Connection object, which establishes the connection to the Human Workflow API and can provide a WorkflowContext for the corresponding admin user.

package com.opitzconsulting.bpm.connection;

import java.util.Map;

import oracle.bpel.services.workflow.client.IWorkflowServiceClient;
import oracle.bpel.services.workflow.client.IWorkflowServiceClientConstants.CONNECTION_PROPERTY;
import oracle.bpel.services.workflow.verification.IWorkflowContext;
import oracle.bpm.client.BPMServiceClientFactory;
import oracle.security.jps.service.credstore.PasswordCredential;

public class HumanWorkflowApiConnection {

  private IWorkflowServiceClient workflowServiceClient;

  public HumanWorkflowApiConnection(Map<CONNECTION_PROPERTY, String> pProperties) {
    final BPMServiceClientFactory bpmServiceClientFactory = BPMServiceClientFactory.getInstance(pProperties, null, null);
    workflowServiceClient = bpmServiceClientFactory.getWorkflowServiceClient();
  }

  public IWorkflowServiceClient getWorkflowServiceClient() {
    return workflowServiceClient;
  }

  public IWorkflowContext createWorkflowContextForAdmin(String pCsfMapname, String pCsfKey) {

    final PasswordCredential passwordCredential = CsfAccessor.readCredentialsfromCsf(pCsfMapname, pCsfKey);

    try {
      return workflowServiceClient.getTaskQueryService().authenticate(passwordCredential.getName(),
      passwordCredential.getPassword(), "jazn.com");
    } catch (Exception e) {
      throw new RuntimeException(String.format("Exception while authenticating Admin User [%s]", passwordCredential.getName()), e);
    }
  }
}

Links:

  1. http://docs.oracle.com/cd/E28280_01/apirefs.1111/e10660/toc.htm
  2. http://docs.oracle.com/cd/E21764_01/core.1111/e10043/csfadmin.htm
  3. http://www.redheap.com/2013/06/secure-credentials-in-adf-application.html

weblogic.server.ServerLifecycleException: Cannot get to the relevant ServerRuntimeMBean for server MGSRV

2. September 2013 1 Kommentar

Lately, I got the following error:

weblogic.server.ServerLifecycleException: Can not get to the relevant ServerRuntimeMBean for server MGSRV.

… weblogic.management.scripting.ScriptException: Error occured while performing shutdown : Error shutting down the server : Can not get to the relevant ServerRuntimeMBean for server MGSRV.

Use dumpStack() to view the full stacktrace
It’s happened, when I wanted to shutdown a managed server via WebLogic Server script “stopManagedWebLogic.sh”. In addition, nodemanger-utility was not able to shutdown managed server via administration console.

After review the issue, I faced to an Oracle Document:
WebLogic Managed Server shutdown failing when using SSL t3s Admin Server address as ADMIN_URL (Doc ID 851065.1)
The main reason for this issue is: “JMX clients using secure protocols were not able to invoke operations on MBeans registered in the Domain Runtime MBeanServer of the AdminServer. Authentication of JMX clients was not correctly performed.”

It seems that is a bug (Bug 8359946):

“Not able to shutdown the managed serves using t3s protocol in the WLST scripts. The same script with t3 protocol is working fine. Adminserver is shutting down fine with either protocol. Not able to shutdown the managed server using WLST. Getting, connecting to successfully connected to Admin Server ‘adminserver’ that belongs to domain…”

You can test it with following steps:

1) Create a domain for WLS 10.3 with 1 Admin and 1 Managed server.

2) Enable SSL and configure the Demo Identity and Demo trust for the Keystores on both the Admin and Managed server.set ADMIN_URL=t3s://localhost:7002

And add the -Dweblogic.security.TrustKeyStore=DemoTrust

4) Start the Admin server.

5) Start the Managed server.

6) Stop the Managed server. Then: Error

What is the solution?
Quick and dirty solution:
You can kill managed server with kill -9 of MGSRV and using administration console for start/stop with nodemanager.

· Advantage: You have a uniform start/stop method for managed server via administration console.

· Disadvantage: Automatic start/stop of managed server(s) via script(s) and operating system is not possible.

Oracle solution:

The bug is fixed in: 12.1.1, for previous WebLogic Server versions are patches available:

WLS Version Patch Number
10.0.1 Patch 8589531
10.0.2 Patch 8359946
10.3.0 Patch 8359946

Please note that patches are applied per WLS installation and not per domain. That is, if you apply this patch on one WLS installation, then all of the servers from all the domains in that installation will have this patch. On the other hand, if you have a managed server in another machine in a domain (that is, set up with its own WLS installation), you need to install this patch on that other machine as well. Generally, patches can only be applied while the server is not running because WLS locks the needed files while it is running. If, however, you are able to apply a patch while WLS is running, you must restart WLS before the patch will take effect.

Kategorien:Weblogic Server Schlagworte: ,

OSB 11g: Stuck Threads when using inbound database adapter

Using a polling database adapter in a OSB proxy service, one may have noticed the following behaviour in Weblogic server:

  • an exception in the server logs about one or even more stuck threads like this:

<BEA-000337> <[STUCK] ExecuteThread: ’10′ for queue: ‘weblogic.kernel.Default (self-tuning)’ has been busy for “705″ seconds working on the request “weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl@21b9db0″, which is more than the configured time (StuckThreadMaxTime) of “600″ seconds. Stack trace:
Thread-118 “[STUCK] ExecuteThread: ’10′ for queue: ‘weblogic.kernel.Default (self-tuning)’” <alive, suspended, waiting, priority=1, DAEMON> {
— Waiting for notification on: oracle.tip.adapter.db.InboundWork@21b8e86[fat lock]
java.lang.Object.wait(Object.java:???)
oracle.tip.adapter.db.InboundWork.run(InboundWork.java:498)
oracle.tip.adapter.db.inbound.InboundWorkWrapper.run(InboundWorkWrapper.java:43)
weblogic.work.ContextWrap.run(ContextWrap.java:39)
weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:528)
weblogic.work.ExecuteThread.execute(ExecuteThread.java:203)
weblogic.work.ExecuteThread.run(ExecuteThread.java:170)
}>

  • and/or server health state from OSB managed server changing from state “Ok” to state “Warning”

Such a behaviour alerts administrators, thinking that something is wrong with the deployed applications or OSB services.

Looking in the Oracle documentation one can find the information that this is behaviour by design and that it can be ignored. To verify the OSB proxy service’s database adapter as the source for this the proxy service has  to be simply disabled in OSB console. Doing so makes the stuck threads disappear. The behaviour seems strange at the  - so why this?

When defining an inbound database adapter, Weblogic threads are used to perform the polling on events occurring in the defined database. Because OSB is designed to deliver high performance and throughput, a number of threads, which depends on the numberOfThreads property in the adapter’s JCA file, is exclusively reserved for the database adapter to perform the inbound polling. Such reserved threads will, due to performance reasons, never be released and never be returned to the corresponding thread pool. After the configured thread timeout, which is by default about 600 seconds, the stuck thread behaviour occurs.

Although this is the default behaviour, it is really confusing and could lead into serious problems, if real threading problems occur in deployed applications/services that are not caused by the adapter threads and which will not be noticed and handled in time. So what can be done to get rid of the stuck adapter threads?

The Oracle documentation proposes to define a separate Work manager and configure this with the proxy service transport’s dispatch policy. To do so, the following steps has to be performed:

  • Define a custom global Work manager using weblogic console with the OSB managed server as deployment target


work_manager

  • Configure the new defined Work manager to ignore stuck threads

ignoreStuckThreads_wm_config

  • Configure OSB proxy service transport’s dispatch policy to use the new defined Work manager

OSB_proxy config

Afterwards the stuck threads behaviour caused the OSB proxy service or by its configured inbound database adapter should not show up again.

BEA-003108: Unicast receive error: java.io.EOFException

Technology, KeyWords:

WebLogic Server 10.3, Cluster, Unicast, Exception;JAVA.IO.EOFEXCEPTION

Error Message: <BEA-003108> <Unicast receive error : java.io.EOFException” >

Problem:

If I restart one Managed Server in a Cluster, I get the following error:

<Error> <Cluster> <BEA-003108> <Unicast receive error : java.io.EOFException

java.io.EOFException

at java.io.DataInputStream.readFully(DataInputStream.java:180)

at java.io.DataInputStream.readLong(DataInputStream.java:399)

at java.io.ObjectInputStream$BlockDataInputStream.readLong(ObjectInputStream.java:2799)

at java.io.ObjectInputStream.readLong(ObjectInputStream.java:960)

at weblogic.cluster.HeartbeatMessage.readExternal(HeartbeatMessage.java:55)

Truncated. see log file for complete stacktrace>

Description

Weblogic declaration from: http://docs.oracle.com/cd/E17904_01/apirefs.1111/e14397/ClusterExtension.html

<BEA-003108> <Unicast receive error : java.io.EOFException”

Error: Unicast receive error: e

An error occurred while trying to receive a message over the cluster broadcast.

Cause

An error occurred while trying to receive a message over the cluster broadcast.

Action

Make sure that the NIC is functioning properly. If you believe there no environment problems exist, contact Oracle Customer Support and provide the stack trace for further analysis.

More detail and Background information:

Note: Sometimes you get Error Message: <BEA-002616> <Failed to listen on channel “Default”…> and it can be a logical effect of error BEA-003108.

The issue ” <BEA-003108> <Unicast receive error : java.io.EOFException”  occurs when using a Unicast cluster and when the cluster debug flags DebugCluster and/or DebugClusterHeartBeats are turned on for some of the managed servers, but not all of them.

The issue also occurs when the debug flags are turned on for the servers that existed previously in the cluster but not on servers which are newly added.

Solution

To resolve this issue, the cluster debug flags need to be enabled consistently: either disabled on all the servers of the cluster or enabled on all of them. In a production environment, debugging should be disabled. Cluster debug flags should be enabled only while debugging a problem with the Unicast cluster.

Solution via AdminServer:

  1. Click on Domain Structure and select your Server
  2. Select debugging tab
  3. Select items, that you want to disable or/enable, e.g. Disable weblogic debugging by click on „disable“(german: „deaktivieren“)
  4. Check Log files, wether the issue is solved.

Refernces

Kategorien:Weblogic Server Schlagworte: , , , ,

BEA-002616: Failed to listen on channel on listenAddress:port

Autor: Mohammad Esad-Djou

Technology, KeyWords:

WebLogic Server 10.3, Cluster, Exception; Too many open files, Socket, Linux, SLES, Error Message: <BEA-002616> <Failed to listen on channel “Default”…>

Problem:

<Critical> <Server> <BEA-002616> <Failed to listen on channel “Default” on IP_NR:Port, failure count: 1, failing for 0 seconds, java.net.SocketException: Too many open files>

Description

Weblogic declaration from: http://docs.oracle.com/cd/E23549_01/apirefs.1111/e14397/Server.html

BEA-002616

Critical: Failed to listen on channel “channel” on listenAddress:port, failure count: fails1, failing for secs2 seconds, e3

  • Description: The server listener will retry the listen after a short delay.
  • Cause: The server got an exception while trying to accept client connections. It will try to backoff to aid recovery.
  • Action: The OS limit for the number of open file descriptor (FD limit) needs to be increased. Tune OS parameters that might help the server to accept more client connections (e.g. TCP accept back log).

More detail and Background information:

“Tune OS parameters” depends on OS which you use. For Linux, kernel parameters need adjustment- exact details depend on the distribution. There are individual user limits for open files. You can use `ulimit -a` command to find out the Linux for the user that owns.

You can find information regarding the important parameters e.g. number of “open files” or similar entry. Please consider, in AIX, the user limits apply as well as a system OS configuration for the total number of open files allowed on the host.

Solution

Here is an example how can you check and isolate the position of error.

1-    Check linux configuration (e.g. on host XXX):

> ulimit –a

core file size          (blocks, -c) 1

data seg size           (kbytes, -d) unlimited

scheduling priority             (-e) 0

file size               (blocks, -f) unlimited

pending signals                 (-i) 127575

max locked memory       (kbytes, -l) 64

max memory size         (kbytes, -m) 13887980

open files                      (-n) 65600 à OK

pipe size            (512 bytes, -p) 8

POSIX message queues     (bytes, -q) 819200

real-time priority              (-r) 0

stack size              (kbytes, -s) 8192

cpu time               (seconds, -t) unlimited

max user processes              (-u) 127575

virtual memory          (kbytes, -v) 17749200

file locks                      (-x) unlimited

The open file descriptor limit is at 65600 as recommended by Oracle.

2-    Check currently open files

  • Find PID of Managed Server

> ps -fea|grep myManagedServer

  • Check open files

Please use lsof (list open files) that lists information about files opened by processes

You see list of open files via (e.g.PID is here 1234)

>lsof -p 1234

In order to findour the number of open files:

> lsof -p 1234 | wc –l

In this case, we observed that application has 13 thousand connections on “CLOSE_WAIT” status. This usually happens when the application doesn’t close the connections properly:

CLOSE_WAIT

The remote end has shut down, waiting for the socket to close.

It’s reason why they are reaching the 65600 limit.

***

After more analyses this case, it is cleared, the issue is a result of weblogic cluster configuration.

“Open files” are waiting to close cluster communication between two nodes. See <BEA-003108> <Unicast receive error: java.io.EOFException” >

Refernces

Kategorien:Weblogic Server Schlagworte: , , ,
Follow

Erhalte jeden neuen Beitrag in deinen Posteingang.