If you are running Oracle SOA suite 11g PS2 (11.1.1.3) and are in production, I salute you. The bad news is that 11.1.1.3 is pretty buggy and quite unstable, and you will run into endless errors. The good news is that the 11.1.1.4 resolves most of these issues.
If you haven't upgraded to 11.1.1.4, then do so immediately to take advantage of its benefits.
Below you will find a list of errors or exceptions you may encounter in 11.1.1.3 that are resolved with 11.1.1.4. Please note that you may still experience the issues below on 11.1.1.4 for other reasons, and don't hesitate to engage Oracle Support when needed. But hopefully most of your issues will get resolved with this patchset, as it did with us.
Good luck!
________________________________________
Issue #1: No free JVM heap space and java.lang.OutOfMemoryError
This error is probably the root cause of most of the issues below. As you deploy more composites to the server, the JVM heap size continues to shrink until you completely run out of memory.
Issue #2: soa_server1 using 99% of the CPU
The soa_server1 java process is pegging the CPU constantly at 99%. This is due to continuous garbage collections that show up in the soa_server1.out output file (if GC logging is enabled).
The output of the "top" command below shows the java process occupying 99.6% of CPU and 59% of memory:
The logs show continuous full garbage collections which hogs the CPU continuously:
When you bring up the "soa_server1" managed server, composites start loading, then once the server runs out of java heap space, all composites become in an "unknown" status, particularly when the number of composites increase.
Issue #4: Unable to retrieve composite details
When logging in to the EM console, when you click on a composite, you may receive the following error.
a. Due to issue #1 above
b. Related to the use of concrete instead of abstract WSDLs when referring to other composites.
Issue #5: WSDL URL is invalid
Composites are inaccessible, and when clicking on the 'Test' button, an HTTP 503 error is returned.
Issue #6: SOA-20003 due to an invalid WSDL when loading composites
When starting up the server, the soa_server1.out log will show the exception below when loading composites (i.e., indicating that it is unable to register the service), and thus the composite becomes in an unknown state.
When starting up the server, the soa_server1.out log will show the exception below when loading composites (i.e., indicating that it is unable to register the service), and thus the composite becomes unavailable.
The logs may show this error occassionally, particularly when the server becomes unstable.
The weblogic.application.ModuleException exception is primarily due to the network adapter not being able to establish the connection.
You may get numerous transaction timeouts after 28 or 48 seconds.
The BEA-149515 error appears for datasources, including many of the product datasources (for example, the AIA datasources).
There is a one-off patch that resolves these issues, but you are better off upgrading to 11.1.1.4 which also includes the fix.
This exception is related to an out-of-the-box datasource and is caused by a cryptic timeout error caused by the resource manager.
This is primarily due to lack of java heap space, which is directly related to issue #1 above.
If you haven't upgraded to 11.1.1.4, then do so immediately to take advantage of its benefits.
Below you will find a list of errors or exceptions you may encounter in 11.1.1.3 that are resolved with 11.1.1.4. Please note that you may still experience the issues below on 11.1.1.4 for other reasons, and don't hesitate to engage Oracle Support when needed. But hopefully most of your issues will get resolved with this patchset, as it did with us.
Good luck!
________________________________________
Issue #1: No free JVM heap space and java.lang.OutOfMemoryError
This error is probably the root cause of most of the issues below. As you deploy more composites to the server, the JVM heap size continues to shrink until you completely run out of memory.
<Jan 19, 2011 3:15:03 PM EST> <Warning> <RMI> <BEA-080003> <RuntimeException thrown by rmi server: javax.management.remote.rmi.RMIConnectionImpl.invoke(Ljavax.management.ObjectName;Ljava.lang.String;Ljava.rmi.MarshalledObject;[Ljava.lang.String;Ljavax.security.auth.Subject;)The WebLogic Administration Console Monitoring Dashboard shows the amount of free java heap space gradually decreasing until it reaches zero:
javax.management.RuntimeErrorException: java.lang.OutOfMemoryError: GC overhead limit exceeded.
javax.management.RuntimeErrorException: java.lang.OutOfMemoryError: GC overhead limit exceeded
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.rethrow(DefaultMBeanServerInterceptor.java:858)
Issue #2: soa_server1 using 99% of the CPU
The soa_server1 java process is pegging the CPU constantly at 99%. This is due to continuous garbage collections that show up in the soa_server1.out output file (if GC logging is enabled).
The output of the "top" command below shows the java process occupying 99.6% of CPU and 59% of memory:
The logs show continuous full garbage collections which hogs the CPU continuously:
1837.993: [Full GC [PSYoungGen: 873856K->800795K(961216K)] [PSOldGen: 3145727K->3145727K(3145728K)] 4019583K->3946523K(4106944K) [PSPermGen: 395967K->395967K(398016K)], 12.8336760 secs] [Times: user=12.59 sys=0.00, real=12.84 secs]Issue #3: Composites in unknown status
1851.327: [Full GC [PSYoungGen: 873856K->809876K(961216K)] [PSOldGen: 3145727K->3145727K(3145728K)] 4019583K->3955604K(4106944K) [PSPermGen: 395986K->395986K(397504K)], 12.7042890 secs] [Times: user=12.59 sys=0.00, real=12.71 secs]
1864.476: [Full GC [PSYoungGen: 873856K->823074K(961216K)] [PSOldGen: 3145727K->3145727K(3145728K)] 4019583K->3968802K(4106944K) [PSPermGen: 396182K->396182K(397440K)], 12.7887330 secs] [Times: user=12.63 sys=0.00, real=12.79 secs]
1877.642: [Full GC [PSYoungGen: 873856K->783438K(961216K)] [PSOldGen: 3145727K->3145721K(3145728K)] 4019583K->3929159K(4106944K) [PSPermGen: 396351K->395549K(396864K)], 16.7096860 secs] [Times: user=16.57 sys=0.00, real=16.71 secs]
When you bring up the "soa_server1" managed server, composites start loading, then once the server runs out of java heap space, all composites become in an "unknown" status, particularly when the number of composites increase.
Issue #4: Unable to retrieve composite details
When logging in to the EM console, when you click on a composite, you may receive the following error.
Unable to retrieve composite details.This error is usually caused by one of two issues:
The composite HelloWorld (1.0) is not available. This could happen because either the composite hsa been undeployed or soa-infra has not yet loaded this composite.
a. Due to issue #1 above
b. Related to the use of concrete instead of abstract WSDLs when referring to other composites.
Issue #5: WSDL URL is invalid
Composites are inaccessible, and when clicking on the 'Test' button, an HTTP 503 error is returned.
Either the WSDL URL is invalid or the WSDL file is not valid or incorrect. - WSDLException: faultCode=OTHER_ERROR: Failed to read WSDL from http://oradev:8001/soa-infra/services/default/HelloWorldComposite/client?WSDL: HTTP connection error code is 503
Issue #6: SOA-20003 due to an invalid WSDL when loading composites
When starting up the server, the soa_server1.out log will show the exception below when loading composites (i.e., indicating that it is unable to register the service), and thus the composite becomes in an unknown state.
<Jan 11, 2011 6:45:59 PM EST> <Error> <oracle.integration.platform> <SOA-20003> <Unable to register service.Issue #7: SOA-20003 and HTTP 503 service unavailable when loading composites
oracle.fabric.common.FabricException: oracle.j2ee.ws.wsdl.LocalizedWSDLException: WSDLException: faultCode=INVALID_WSDL: The document: http://oradev:8001/soa-infra/services/default/HelloWorld/HelloWorld.wsdl is not a wsdl file or does not have a root element of "definitions" in the "http://schemas.xmlsoap.org/wsdl/" namespace or the "http://www.w3.org/2004/08/wsdl" namespace.: WSDLException: faultCode=INVALID_WSDL: The document: http://oradev:8001/soa-infra/services/default/HelloWorld/HelloWorld.wsdl is not a wsdl file or does not have a root element of "definitions" in the "http://schemas.xmlsoap.org/wsdl/" namespace or the "http://www.w3.org/2004/08/wsdl" namespace.
at oracle.fabric.composite.model.CompositeModel.loadImports(CompositeModel.java:272)
When starting up the server, the soa_server1.out log will show the exception below when loading composites (i.e., indicating that it is unable to register the service), and thus the composite becomes unavailable.
<Jan 19, 2011 2:57:56 PM EST> <Error> <oracle.integration.platform> <SOA-20003> <Unable to register service.Issue #8: java.sql.SQLException: The Network Adapter could not establish the connection
oracle.fabric.common.FabricException: Error in getting XML input stream: http://oradev:8001/soa-infra/services/default/HelloWorld/HelloWorld?WSDL: Response: '503: Service Unavailable' for url: 'http://oradev:8001/soa-infra/services/default/HelloWorld/HelloWorld?WSDL'
at oracle.fabric.common.metadata.MetadataManagerImpl.getInputStreamFromAbsoluteURL(MetadataManagerImpl.java:276)
The logs may show this error occassionally, particularly when the server becomes unstable.
####<Jan 11, 2011 6:35:39 PM EST> <Info> <JDBC> <oradev> <soa_server1> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1294788939135> <BEA-001156> <Stack trace associated with message 001129 follows:Issue #9: BEA-149265 due to weblogic.application.ModuleException
java.sql.SQLException: The Network Adapter could not establish the connection
The weblogic.application.ModuleException exception is primarily due to the network adapter not being able to establish the connection.
<Jan 19, 2011 3:05:26 PM EST> <Error> <Deployer> <BEA-149265> <Failure occurred in the execution of deployment request with ID '1295467469460' for task 'weblogic.deploy.configChangeTask.2'. Error is: 'weblogic.application.ModuleException: 'Issue #10: BEA-000000 due to timeout
weblogic.application.ModuleException:
at weblogic.jdbc.module.JDBCModule.prepare(JDBCModule.java:290)
at weblogic.application.internal.flow.ModuleListenerInvoker.prepare(ModuleListenerInvoker.java:199)
at weblogic.application.internal.flow.DeploymentCallbackFlow$1.next(DeploymentCallbackFlow.java:507)
at weblogic.application.utils.StateMachineDriver.nextState(StateMachineDriver.java:41)
at weblogic.application.internal.flow.DeploymentCallbackFlow.prepare(DeploymentCallbackFlow.java:149)
Truncated. see log file for complete stacktrace
Caused By: weblogic.common.ResourceException: weblogic.common.ResourceException: Could not create pool connection. The DBMS driver exception was: The Network Adapter could not establish the connection
You may get numerous transaction timeouts after 28 or 48 seconds.
<Jan 19, 2011 3:15:03 PM EST> <Warning> <oracle.soa.mediator.common> <BEA-000000> <Transaction commit failed due to excetipnIssue #11: BEA-149515 for numerous data sources
weblogic.transaction.RollbackException: Transaction timed out after 48 seconds
BEA1-0BF2AA3AE988E50C6CCB
at weblogic.transaction.internal.TransactionImpl.throwRollbackException(TransactionImpl.java:1871)
The BEA-149515 error appears for datasources, including many of the product datasources (for example, the AIA datasources).
<Jan 19, 2011 3:15:21 PM EST> <Warning> <JMX> <BEA-149515> <An error was encountered getting the attribute DatabaseProductName on the MBean com.bea:ServerRuntime=soa_server1,Name=AIADataSource,Type=JDBCDataSourceRuntime during a call to getAttributes>Issue #12: Multiple WSM errors
There is a one-off patch that resolves these issues, but you are better off upgrading to 11.1.1.4 which also includes the fix.
<Jan 19, 2011 3:15:24 PM EST> <Error> <oracle.wsm.resources.security> <WSM-00006> <Error in receiving the request: oracle.wsm.security.SecurityException: WSM-00008 : Web service authentication failed..>Issue #13: XAResource.XAER_RMERR start() failed on resource 'EDNDataSource_soa_domain'
<Jan 19, 2011 3:15:24 PM EST> <Error> <oracle.wsm.resources.enforcement> <WSM-07607> <Failure in execution of assertion {http://schemas.oracle.com/ws/2006/01/securitypolicy}wss-username-token executor class oracle.wsm.security.policy.scenario.executor.WssUsernameTokenScenarioExecutor.>
<Jan 19, 2011 3:15:24 PM EST> <Error> <oracle.wsm.resources.enforcement> <WSM-07602> <Failure in WS-Policy Execution due to exception.>
<Jan 19, 2011 3:15:24 PM EST> <Error> <oracle.wsm.resources.enforcement> <WSM-07501> <Failure in Oracle WSM Agent processRequest, category=security, function=agent.function.service, application=soa-infra, composite=HelloWorldComposite, modelObj=client, policy=oracle/wss_username_token_service_policy, policyVersion=1, assertionName={http://schemas.oracle.com/ws/2006/01/securitypolicy}wss-username-token.>
<Jan 19, 2011 3:15:24 PM EST> <Error> <oracle.webservices.service> <OWS-04115> <An error occurred for port: FabricProvider: oracle.fabric.common.PolicyEnforcementException: FailedAuthentication : The security token cannot be authenticated..>
<Jan 19, 2011 3:15:24 PM EST> <Error> <oracle.wsm.resources.security> <WSM-00069> <The security header is missing.>
This exception is related to an out-of-the-box datasource and is caused by a cryptic timeout error caused by the resource manager.
<Jan 18, 2011 4:53:49 PM EST> <Warning> <oracle.integration.platform.blocks.event.saq> <SOA-31013> <Error handling message (rolling back).java.sql.SQLException: Unexpected exception while enlisting XAConnection java.sql.SQLException: XA error: XAResource.XAER_RMERR start() failed on resource 'EDNDataSource_soa_domain': XAER_RMERR : A resource manager error has occured in the transaction branchweblogic.transaction.internal.ResourceAccessException: Transaction has timed out when making request to XAResource 'EDNDataSource_soa_domain'.Issue #14: Numerous BEA-000000 errors
This is primarily due to lack of java heap space, which is directly related to issue #1 above.
<Jan 13, 2011 1:59:27 PM EST> <Warning> <oracle.soa.adapter> <BEA-000000> <JMSAdapter MyAdapterPoll JMSMessageConsumer_init: Retrying connection; attempt #1415>
<Jan 13, 2011 1:59:27 PM EST> <Warning> <oracle.soa.adapter> <BEA-000000> <JMSAdapter MyAdapterPoll
BINDING.JCA-12135
ERRJMS_ERR_CR_QUEUE_CONS.
ERRJMS_ERR_CR_QUEUE_CONS.
Unable to create Queue consumer due to JMSException.
Please examine the log file to determine the problem.
at oracle.tip.adapter.jms.JMS.JMSConnection.createConsumer(JMSConnection.java:620)
No comments:
Post a Comment