Thursday, October 20, 2016

AWR Snapshots are not generated Automatically

Scenario:
------------

Oracle Database 11.2.0.4. My AWR Snapshots were not being generated even though it was full enabled.

Basic Checks done:
------------------------

1. Checked that I had enough free space on SYSAUX tablespace. I was using only 5GB of the total 20GB.
2. System Parameter statistics_level was TYPICAL.
3. DBMS_STATS.GET_STATS_HISTORY_RETENTION was to 30 days.
4. The snapshot Interval was 60 minutes and Retention was 8 days.

Query Check the AWR intervals:
=========================
select
extract( day from snap_interval) *24*60+
extract( hour from snap_interval) *60+
extract( minute from snap_interval ) "Snapshot Interval",
extract( day from retention) *24*60+
extract( hour from retention) *60+
extract( minute from retention ) "Retention Interval"
from
dba_hist_wr_control;

5. Everything was OK,So finally checked the process responsible for AWR collection ie) MMON

Now MMON was not running.

Now it is a critical system and CRP testing is going, So i cannot bring down the system but i need to spawn the MMON process.

Here is the workaround :
-------------------------------

Just enable and disable the restricted sessions.

SQL> alter system enable restricted session;

System altered.

SQL> alter system disable restricted session;

System altered.

Now check the MMON.

[oradoyen@saravanadb trace]$ ps -ef|grep mmon
oradoy  9732     1  3 18:46 ?        00:00:00 ora_mmon_ORADOY
oradoy  9977 50953  0 18:47 pts/8    00:00:00 grep mmon
oradoy 28182     1  0 Oct10 ?        00:11:40 ora_mmon_DEV

Now AWR snapshots started generating.

Note :
Actually this is a bug in 11.2.0.4 and it is fixed by a patch 19565533.

Note id : 2023652.1 AWR Snapshots Are Not Being Created Because MMON Is Not Being Respawned


Thursday, October 6, 2016

ORA-20100: Temporary file creation for FND_FILE failed

Scenario:
-----------

While running a report in Payables or any of the program errored out and the log file shows the following error in R12.2.4

ORA-20100: Temporary file creation for FND_FILE failed

Error in Log file :
----------------------

Enter Password:
MSG-00101: Error occured in BATCH_APPROVAL()
MSG-00102: Error Code : -20100
MSG-00103: Error Message : ORA-20100: ORA-20100: Temporary file creation for FND_FILE failed.
Directory &FILE_DIR is invalid, contact your system administrator.

Action: Check that the directory is a valid directory with write permissions on the database server node and is listed under utl_file_dir para
ORA-06512: at "APPS.FND_FILE", line 319
ORA-06512: at "APPS.FND_FILE", line 364
ORA-06512: at "APPS.FND_FILE", line 421
ORA-06512: at "APPS.AP_APPROVAL_PKG", line 10672
MSG-00000: When Others:User-Defined Exception
REP-1419: 'beforereport': PL/SQL program aborted.
REP-0069: Internal error
REP-57054: In-process job terminated:Terminated with error:
REP-1419: MSG-00101: Error occured in BATCH_APPROVAL()
MSG-00102: Error Code : -20100
MSG-00103: Error Message : ORA-20100: ORA-20100: Temporary file creation for FND_FILE failed.
Directory &FILE_DIR is invalid, contact your system administrator.
Action: Check that the directory is a valid directory with write permissions on the database server node and is listed under utl_file_dir para
ORA-06512: at "APPS.FND_FILE", lin

Solution:
-----------

1. Check the APPLPTMP for ex. /usr/tmp/SARA
2. check whether that exsists in utl_file_directory.
Note : Better to check with gv$ view like below
select name,value FROM gv$parameter2 WHERE name='utl_file_dir' and value like '%SAR%';

If it is not same which is in APPLPTMP. then

a) Change the APPLPTMP location to any one location which is there in UTL file in env file and XML and bounce the concurrent
or
b) Add the same value which is in APPLPTMP to the UTL file location and bounce the db.






Tuesday, September 13, 2016

XML-20201: (Fatal Error) Expected name instead of .

Error :
-------

Output post processor failed with below error processing report output

Caused by: org.xml.sax.SAXParseException: : XML-20201: (Fatal Error) Expected name instead of  .

Concurrent Log file Error:
-------------------------------

Inserting invoice IDs into Temp Table
SQL STATEMENT :  INSERT into xxdn.xxdn_ACCOUNT_ANALYSIS_INVTEMP  SELECT pv.VENDOR_NAME supplier_name,   ai.invoice_num invoice_number,   ab.INVOICE_DATE invoice_date,   ab.invoice_amount  invoice_amount,   ab.invoice_id SOURCE_INVOICE_ID,   ab.ORG_ID,   ab.SET_OF_BOOKS_ID   FROM   hr_organization_information oi, hr_all_organization_units_tl ae, AND ae.language = USERENV('LANG')   AND NVL(ORG_ID,85 ) = 85

Inserted records into Temp Table successfully..

+------------- 1) PUBLISH -------------+
Beginning post-processing of request 321496451 on node USWS1HMSDVCM02 at 12-SEP-2016 16:48:24.
Post-processing of request 321496451 failed at 12-SEP-2016 16:48:28 with the error message:
One or more post-processing actions failed. Consult the OPP service log for details.

OPP Log file shows the following error:
--------------------------------------------------
[9/12/16 4:48:28 PM] [UNEXPECTED] [2058679:RT321496451] java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at oracle.apps.xdo.template.FOProcessor.generate(FOProcessor.java:975)
        at oracle.apps.xdo.oa.schema.server.TemplateHelper.runProcessTemplate(TemplateHelper.java:5943)
        at oracle.apps.xdo.oa.schema.server.TemplateHelper.processTemplate(TemplateHelper.java:3459)
        at oracle.apps.xdo.oa.schema.server.TemplateHelper.processTemplate(TemplateHelper.java:3548)
        at oracle.apps.fnd.cp.opp.XMLPublisherProcessor.process(XMLPublisherProcessor.java:290)
        at oracle.apps.fnd.cp.opp.OPPRequestThread.run(OPPRequestThread.java:157)
Caused by: org.xml.sax.SAXParseException: : XML-20201: (Fatal Error) Expected name instead of  .
        at oracle.xdo.parser.v2.XMLError.flushErrorHandler(XMLError.java:441)
        at oracle.xdo.parser.v2.XMLError.flushErrors1(XMLError.java:303)
        at oracle.xdo.parser.v2.XMLReader.scanNameChars(XMLReader.java:1061)
        at oracle.xdo.parser.v2.XMLReader.scanQName(XMLReader.java:1871)
        at oracle.xdo.parser.v2.XMLReader.getEntity(XMLReader.java:2125)
        at oracle.xdo.parser.v2.NonValidatingParser.parseContentEntityRef(NonValidatingParser.java:1654)
        at oracle.xdo.parser.v2.NonValidatingParser.parseElement(NonValidatingParser.java:1272)
        at oracle.xdo.parser.v2.NonValidatingParser.parseRootElement(NonValidatingParser.java:338)
        at oracle.xdo.parser.v2.NonValidatingParser.parseDocument(NonValidatingParser.java:285)
        at oracle.xdo.parser.v2.XMLParser.parse(XMLParser.java:289)
        ... 17 more


Solution:
-----------

select invoice_number from xxdn_ACCOUNT_ANALYSIS_INVTEMP where invoice_number like '%&%';

The above table is taken from the output file of the concurrent request and this gives the output column in which the "&" is included in the value which is supposed to be removed.

Checking output xml file Line 17018, we notice “&”, which is causing the issue.
2016-0402 & 2016-5829

You will have to run a data fix for this particular output or check with the developers to fix this.

Thursday, April 7, 2016

OPM:Can not find one alive process, Page cannot be displayed in 11i after running ADCFGCLONE

Issue :

 After running adcfgclone in apps Tier in 11i , It got completed without any error and When we tried to login the First page comes but when we try to click on homepage, it redirects to blank page which says "Page cannot be displayed" instead of login page.

In Apache Error log:
--------------------------

iAS Home/Apache/Apache/logs :

[Tue Apr  5 20:41:43 2016] [error] [client 10.1.163.122] File does not exist:
[Tue Apr  5 20:45:05 2016] [error] [client 10.1.163.122] File does not exist: /u01/app/doyen/doyencomn/html/US/ICXINDEX_doyen_usws1rdvcm04.htm
[Tue Apr  5 20:45:22 2016] [error] OPM:Can not find one alive process
[Tue Apr  5 20:45:22 2016] [error] [client 10.1.163.122] File does not exist: /servlets/weboam/oam/oamLogin/oam/oamLogin
[Tue Apr  5 20:51:13 2016] [error] OPM: EW: Fail to start process with mod=JServ and grp=DiscoGroup, it's possible that your configuration file is not correct.
[Tue Apr  5 20:51:13 2016] [error] OPM: EW: Fail to start process with mod=JServ and grp=OACoreGroup, it's possible that your configuration file is not correct.
[Tue Apr  5 20:51:13 2016] [error] OPM: EW: Fail to start process with mod=JServ and grp=OACoreGroup, it's possible that your configuration file is
[Tue Apr  5 21:12:18 2016] [notice] Oracle HTTP Server Powered by Apache/1.3.19 configured -- resuming normal operations
[Tue Apr  5 21:25:42 2016] [error] [client 10.1.163.122] File does not exist: /u01/app/doyen/doyencomn/portal/doyen_doyendb02/favicon.ico
[Tue Apr  5 21:25:45 2016] [error] [client 10.1.163.122] File does not exist: /u01/app/doyen/doyencomn/portal/doyen_doyendb02/favicon.ico
[Tue Apr  5 21:25:45 2016] [error] OPM:Can not find one alive process

In Jserv Logs:
----------------

[apdoyen@doyendb02 jvm]$ pwd
/u01/app/doyen/doyenora/iAS/Apache/Jserv/logs/jvm

vi OACoreGroup.4.stderr

Please use CMSClassUnloadingEnabled in place of CMSPermGenSweepingEnabled in the future
Could not create the Java virtual machine.

vi XmlSvcsGrp.0.stdout

/usr/bin/jdk1.6.0_30/jre/bin/java -DCLIENT_PROCESSID=8314 -verbose:gc -Xmx1600M -Xms1600M -XX:MaxPermSize=128M -XX:NewRatio=2 -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:
+PrintClassHistogram -XX:+UseTLAB -XX:+CMSPermGenSweepingEnabled -XX:+CMSClassUnloadingEnabled -Djava.awt.headless=false -DFND_TOP=/u01/app/doyen/doyenappl/fnd/11.5.0 -OA_HTML=/u01/app/doyen/doyencomn/html/ -DOA_MEDIA=/u01/app/doyen/doyencomn/java/oracle/apps/media/ -DOASMTPServer=doyendb02.hrbl.net -DWebProxyHost=doyendb02.hrbl.net -
DWebProxyPort=80 -DWebProxyByPassDomain=hrbl.net -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.ssl=false -
DOASSLCACertFile=/u01/app/doyen/doyenora/iAS/Apache/Apache/conf/ssl.crt/ca-bundle.crt -DOXTAInPoolSize=1 -DOXTAOutThreads=1 -DOXTAOutUseProxy=false -
DOXTAOutProxyHost=doyendb02.hrbl.net -DOXTAOutProxyPort=80 -DCOMMON_TOP=/u01/app/doyen/doyencomn -DOXTALogDebugMsg=false -DEXTERNAL_URL=http://doyendb02.hrbl.net:8016 -
Djbo.323.compatible=true -DAPPLRGF=/u01/app/doyen/doyencomn/rgf/doyen_doyendb02 -DJTFDBCFILE=/u01/app/doyen/doyenappl/fnd/11.5.0/secure/doyenapps01_doyen.dbc -
Dservice.Logging.common.filename=/u01/app/doyen/doyencomn/temp/ibe.log -Dframework.Logging.system.filename=/u01/app/doyen/doyencomn/temp/fwsys.log -
DIMT_COM_PROPERTY_FILE=/u01/app/doyen/doyenappl/imt/11.5.0/admin/scripts/imtjserv.properties -Dpoolsize=100 -Dminpoolsize=10 -Dpoolincrement=10 -Dpooldelayincrement=3 -
DBNEDBCFILE=/u01/app/doyen/doyenappl/fnd/11.5.0/secure/doyenapps01_doyen.dbc -Djserv.session.getValue.instrument=false org.apache.jserv.JServ -opmpropfile
/u01/app/doyen/doyenora/iAS/Apache/Jserv/etc/xmlsvcs.properties -opmhost doyendb02.hrbl.net -opmport 8116 -opmgrp XmlSvcsGrp -opmindex 0 -opmprocid 7
-------------------------------
Error occurred during initialization of VM
Could not reserve enough space for object heap
---------------------------------------------------

Solution  :
-----------
Cause :
---------

System was unable to reserve 1600M java heap memory for the jvm …

Solution : 

Reduced the java heap memory to be used by the jvm to 512M (updated the s_jvm_options  in the CONTEXT_FILE and ran autoconfig)


Before:
-verbose:gc –Xmx1600M –Xms1600M -XX:MaxPermSize=128M -XX:NewRatio=2 -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:
+PrintClassHistogram -XX:+UseTLAB -XX:+CMSClassUnloadingEnabled

After:

-verbose:gc -Xmx512M -Xms512M -XX:MaxPermSize=128M -XX:NewRatio=2 -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -
XX:+PrintClassHistogram -XX:+UseTLAB -XX:+CMSClassUnloadingEnabled

After this Run Autoconfig and retest the issue.

Reference : Error 'Could not reserve enough space for object heap' During Initialization of JVM (Doc ID 1028915.1)

Tuesday, March 29, 2016

Cause: The field PARAMETER.CONFIG could not be located or read in R12.2.4

Problem Summary
------------------------

In R12.2.4 After clone , the concurrent ouput/log file shows the error Cannot read value from field PARAMETER.CONFIG


Problem Description
-------------------------

After clone we get the following error in log and output file , but the request completed Normal


Error in the Output/log file :
---------------------------------

Cause: The field PARAMETER.CONFIG could not be located or read.

Action: This error is normally the result of an incorrectly-entered field name string in a trigger, or a field name string that does not uniquely specify a field in your form.

Correct your trigger logic to precisely specify a valid field.


Env Details:
--------------

2 DB node , 2 CM nodes , 2 APP nodes


To find out the issue :
-------------------------

Enable Trace for FNDFS in all the nodes (CM) in listener file :
--------------------------------------------------------------------------

( SID_DESC = ( SID_NAME = FNDFS  )
                 ( ORACLE_HOME = /u01/app/doyuat/fs1/EBSapps/10.1.2 )
                 ( PROGRAM = /u01/app/DOYuat/fs1/EBSapps/appl/fnd/12.0.0/bin/FNDFS )
                 (

envs='EPC_DISABLED=TRUE,NLS_LANG=American_America.UTF8,LD_LIBRARY_PATH=/u01/app/DOYuat/fs1/EBSapps/10.1.2/lib32:/u01/app/DOYuat/fs1/EBSapps/10.1.2/lib:/usr/X11R6/lib:/usr/openwin/lib:/u01/app/DOYuat/fs1/EBSapps/10.1.2/jdk/jre/lib/i386:/u01/app/DOYuat/fs1/EBSapps/10.1.2/jdk/jre/lib/i386/server:/u01/app/DOYuat/fs1/EBSapps/10.1.2/jdk/jre/lib/i386/native_threads:/u01/app/DOYuat/fs1/EBSapps/appl/sht/12.0.0/lib,SHLIB_PATH=/u01/app/DOYuat/fs1/EBSapps/10.1.2/lib32:/u01/app/DOYuat/fs1/EBSapps/10.1.2/lib:/usr/X11R6/lib:/usr/openwin/lib:/u01/app/DOYuat/fs1/EBSapps/10.1.2/jdk/jre/lib/i386:/u01/app/DOYuat/fs1/EBSapps/10.1.2/jdk/jre/lib/i386/server:/u01/app/DOYuat/fs1/EBSapps/10.1.2/jdk/jre/lib/i386/native_threads:/u01/app/DOYuat/fs1/EBSapps/appl/sht/12.0.0/lib,LIBPATH=/u01/app/DOYuat/fs1/EBSapps/10.1.2/lib32:/u01/app/DOYuat/fs1/EBSapps/10.1.2/lib:/usr/X11R6/lib:/usr/openwin/lib:/u01/app/DOYuat/fs1/EBSapps/10.1.2/jdk/jre/lib/i386:/u01/app/DOYuat/fs1/EBSapps/10.1.2/jdk/jre/lib/i386/server:/u01/app/DOYuat/fs1/EBSapps/10.1.2/jdk/jre/lib/i386/native_threads:/u01/app/DOYuat/fs1/EBSapps/appl/sht/12.0.0/lib,APPLFSTT=DOYUAT_806_BALANCE;DOYUATB_FO;DOYUATB;DOYUAT;DOYUATA;DOYUAT_FO;DOYUATA_FO,APPLFSWD=/u01/app/DOYuat/fs1/inst/apps/DOYUAT_doyensys01/appl/admin;/u01/app/DOYuat/fs1/inst/apps/DOYUAT_doyensys01/appltmp;/u01/app/DOYuat/fs1/FMW_Home/Oracle_EBS-app1/applications/oacore/html/oam/nonUix/launchMode/restricted,FNDFS_LOGGING=ON,FNDFS_LOGFILE=/u01/app/DOYuat/fs1/inst/apps/DOYUAT_doyensys01/logs/ora/10.1.2/network/FNDFS_DOYuat.log' )))


Line to be added in the listener for enabling trace and path of the log file :
---------------------------------------------------------------------------

FNDFS_LOGGING=ON,FNDFS_LOGFILE=/u01/app/doyuat/fs1/inst/apps/DOYUAT_doyensys01/logs/ora/10.1.2/network/FNDFS_doyuat.log'

Now bounce the listener and check for the request log/output files.

a) Retest the issue .

b) You will get the FNDFS log output in the specified location as mentioned in the listener (In the node in which the particular request try to fetch log/output file).

c) Now the error in the log file shows when connecting to the Node using the TWO_TASK it errored out.


Solution :
-----------

1. Check the TWO_TASK in Application nodes.(ex.It can be DOYUAT_BALANCE)
2. Check the listener file entry in Application Tier for the parameter APPLFSTT for which the TWO_TASK is included.
3. The same two_task entry should be included in both the CM nodes APPLFSTT value in listener file.
4. The Two_TASK value for CM nodes may be different like DOYUAT1 and DOYUAT2 for each nodes. But listener file should have included the Application's TWO_TASK for APPLFSTT.
5. The same value should be added in TNSNAMES file in both the CM NODES through which the Application node will connect to the CM node.
6. As a permenent fix change the value of APPLFSTT in XML file and run Autoconfig in CM node.


Following tns entry needs to be added in the tnsnames.ora in Both CM Nodes :
-------------------------------------------------------------------------------

DOYUAT_BALANCE=
        (DESCRIPTION=
            (ADDRESS_LIST=
                (LOAD_BALANCE=YES)
                (FAILOVER=YES)
                (ADDRESS=(PROTOCOL=tcp)(HOST=doyensys02.doyen.net)(PORT=1526))
                (ADDRESS=(PROTOCOL=tcp)(HOST=doyensys01.doyen.net)(PORT=1526))
            )
            (CONNECT_DATA=
                (SERVICE_NAME=DOYUAT)
            )
        )


After making the Entry in parameter (APPLFSTT)  in listener file in Both CM nodes  :


( SID_DESC = ( SID_NAME = FNDFS  )
                 ( ORACLE_HOME = /u01/app/doyuat/fs1/EBSapps/10.1.2 )
                 ( PROGRAM = /u01/app/DOYuat/fs1/EBSapps/appl/fnd/12.0.0/bin/FNDFS )
                 (envs='EPC_DISABLED=TRUE,NLS_LANG=American_America.UTF8,LD_LIBRARY_PATH=/u01/app/DOYuat/fs1/EBSapps/10.1.2/lib32:/u01/app/DOYuat/fs1/EBSapps/10.1.2/lib:/usr/X11R6/lib:/usr/openwin/lib:/u01/app/DOYuat/fs1/EBSapps/10.1.2/jdk/jre/lib/i386:/u01/app/DOYuat/fs1/EBSapps/10.1.2/jdk/jre/lib/i386/server:/u01/app/DOYuat/fs1/EBSapps/10.1.2/jdk/jre/lib/i386/native_threads:/u01/app/DOYuat/fs1/EBSapps/appl/sht/12.0.0/lib,SHLIB_PATH=/u01/app/DOYuat/fs1/EBSapps/10.1.2/lib32:/u01/app/DOYuat/fs1/EBSapps/10.1.2/lib:/usr/X11R6/lib:/usr/openwin/lib:/u01/app/DOYuat/fs1/EBSapps/10.1.2/jdk/jre/lib/i386:/u01/app/DOYuat/fs1/EBSapps/10.1.2/jdk/jre/lib/i386/server:/u01/app/DOYuat/fs1/EBSapps/10.1.2/jdk/jre/lib/i386/native_threads:/u01/app/DOYuat/fs1/EBSapps/appl/sht/12.0.0/lib,LIBPATH=/u01/app/DOYuat/fs1/EBSapps/10.1.2/lib32:/u01/app/DOYuat/fs1/EBSapps/10.1.2/lib:/usr/X11R6/lib:/usr/openwin/lib:/u01/app/DOYuat/fs1/EBSapps/10.1.2/jdk/jre/lib/i386:/u01/app/DOYuat/fs1/EBSapps/10.1.2/jdk/jre/lib/i386/server:/u01/app/DOYuat/fs1/EBSapps/10.1.2/jdk/jre/lib/i386/native_threads:/u01/app/DOYuat/fs1/EBSapps/appl/sht/12.0.0/lib,APPLFSTT=DOYUAT_BALANCE;DOYUAT_806_BALANCE;DOYUATB_FO;DOYUATB;DOYUAT;DOYUATA;DOYUAT_FO;DOYUATA_FO,APPLFSWD=/u01/app/DOYuat/fs1/inst/apps/DOYUAT_doyensys01/appl/admin;/u01/app/DOYuat/fs1/inst/apps/DOYUAT_doyensys01/appltmp;/u01/app/DOYuat/fs1/FMW_Home/Oracle_EBS-app1/applications/oacore/html/oam/nonUix/launchMode/restricted,FNDFS_LOGGING=ON,FNDFS_LOGFILE=/u01/app/DOYuat/fs1/inst/apps/DOYUAT_doyensys01/logs/ora/10.1.2/network/FNDFS_DOYuat.log' )   )  )


Monday, July 20, 2015

ORA-02085: database link DOYEN_REMOTEDB.DOYENSYS@USERNAME connects to DOYEN_REMOTEDB

ERROR:
=====

ORA-02085: database link DOYEN_REMOTEDB.DOYENSYS@USERNAME connects to DOYEN_REMOTEDB

SCENARIO:
========

SQL> select * from dual@DOYEN_REMOTEDB@USERNAME;
select * from dual@DOYEN_REMOTEDB@USERNAME

ERROR in líne 1:

ORA-02085: database link DOYEN_REMOTEDB.DOYENSYS@USERNAME connects to DOYEN_REMOTEDB

REASON :
=======

Global name in the database connecting to the remote is set to TRUE.

select name, value from v$parameter where name in ('db_name', 'db_domain', 'global_names');
NAME                 VALUE
-------------------- ----------------------------------------
db_domain
global_names         TRUE
db_name              mydb

SOLUTION :
=========

Simply at local db set

alter system set global_Names = FALSE;

Now it will work fine.

Wednesday, May 27, 2015

ORA-04031: unable to allocate bytes of shared memory ("","","","")


Problem Description:
===============

Archive log has been shipped from primary but not able to apply in standby.


Error in DB Alert log:
================

Completed Media Recovery
Checking to start in-flux buffer recovery from SCN 1504.3738248416 to SCN  (non-inclusive) 1504.3738248416

ORA-00283: recovery session canceled due to errors
ORA-01237: cannot extend datafile 113
ORA-01110: data file 113: '+DATA/doyensys_p/datafile/sysaux.408.874229253'
ORA-17505: ksfdrsz:1 Failed to resize file to size 1126400 blocks
ORA-00569: Failed to acquire global enqueue.
Slave exiting with ORA-283 exception
ORA-00283: recovery session canceled due to errors
ORA-01237: cannot extend datafile 113
ORA-01110: data file 113: '+DATA/doyensys_p/datafile/sysaux.408.874229253'
ORA-17505: ksfdrsz:1 Failed to resize file to size 1126400 blocks
ORA-00569: Failed to acquire global enqueue.


When we checked for the size, We have enough size in the Disk. But still it shows the above error. So we checked for the asm log.


Asm alert log :
===========

Wed May 20 19:55:29 2015
Errors in file /u01/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_ora_95265.trc  (incident=353797):
ORA-04031: unable to allocate  bytes of shared memory ("","","","")
Incident details in: /u01/app/oracle/diag/asm/+asm/+ASM1/incident/incdir_353797/+ASM1_ora_95265_i353797.trc
Wed May 20 19:55:30 2015

Solution:
========

There was some memory issue happened in the asm instance though it has the enough memory.So we have just bounced the asm instance.

1. Recovery needs to be cancelled, But it has already been done.
2. shut down the database
3. connect to asm instance
4. srvctl stop asm -n nodename
5. srvctl start asm -n nodename
6. Now start the database in mount and started the recovery.