Tuesday, September 13, 2016

XML-20201: (Fatal Error) Expected name instead of .

Error :
-------

Output post processor failed with below error processing report output

Caused by: org.xml.sax.SAXParseException: : XML-20201: (Fatal Error) Expected name instead of  .

Concurrent Log file Error:
-------------------------------

Inserting invoice IDs into Temp Table
SQL STATEMENT :  INSERT into xxdn.xxdn_ACCOUNT_ANALYSIS_INVTEMP  SELECT pv.VENDOR_NAME supplier_name,   ai.invoice_num invoice_number,   ab.INVOICE_DATE invoice_date,   ab.invoice_amount  invoice_amount,   ab.invoice_id SOURCE_INVOICE_ID,   ab.ORG_ID,   ab.SET_OF_BOOKS_ID   FROM   hr_organization_information oi, hr_all_organization_units_tl ae, AND ae.language = USERENV('LANG')   AND NVL(ORG_ID,85 ) = 85

Inserted records into Temp Table successfully..

+------------- 1) PUBLISH -------------+
Beginning post-processing of request 321496451 on node USWS1HMSDVCM02 at 12-SEP-2016 16:48:24.
Post-processing of request 321496451 failed at 12-SEP-2016 16:48:28 with the error message:
One or more post-processing actions failed. Consult the OPP service log for details.

OPP Log file shows the following error:
--------------------------------------------------
[9/12/16 4:48:28 PM] [UNEXPECTED] [2058679:RT321496451] java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at oracle.apps.xdo.template.FOProcessor.generate(FOProcessor.java:975)
        at oracle.apps.xdo.oa.schema.server.TemplateHelper.runProcessTemplate(TemplateHelper.java:5943)
        at oracle.apps.xdo.oa.schema.server.TemplateHelper.processTemplate(TemplateHelper.java:3459)
        at oracle.apps.xdo.oa.schema.server.TemplateHelper.processTemplate(TemplateHelper.java:3548)
        at oracle.apps.fnd.cp.opp.XMLPublisherProcessor.process(XMLPublisherProcessor.java:290)
        at oracle.apps.fnd.cp.opp.OPPRequestThread.run(OPPRequestThread.java:157)
Caused by: org.xml.sax.SAXParseException: : XML-20201: (Fatal Error) Expected name instead of  .
        at oracle.xdo.parser.v2.XMLError.flushErrorHandler(XMLError.java:441)
        at oracle.xdo.parser.v2.XMLError.flushErrors1(XMLError.java:303)
        at oracle.xdo.parser.v2.XMLReader.scanNameChars(XMLReader.java:1061)
        at oracle.xdo.parser.v2.XMLReader.scanQName(XMLReader.java:1871)
        at oracle.xdo.parser.v2.XMLReader.getEntity(XMLReader.java:2125)
        at oracle.xdo.parser.v2.NonValidatingParser.parseContentEntityRef(NonValidatingParser.java:1654)
        at oracle.xdo.parser.v2.NonValidatingParser.parseElement(NonValidatingParser.java:1272)
        at oracle.xdo.parser.v2.NonValidatingParser.parseRootElement(NonValidatingParser.java:338)
        at oracle.xdo.parser.v2.NonValidatingParser.parseDocument(NonValidatingParser.java:285)
        at oracle.xdo.parser.v2.XMLParser.parse(XMLParser.java:289)
        ... 17 more


Solution:
-----------

select invoice_number from xxdn_ACCOUNT_ANALYSIS_INVTEMP where invoice_number like '%&%';

The above table is taken from the output file of the concurrent request and this gives the output column in which the "&" is included in the value which is supposed to be removed.

Checking output xml file Line 17018, we notice “&”, which is causing the issue.
2016-0402 & 2016-5829

You will have to run a data fix for this particular output or check with the developers to fix this.

Thursday, April 7, 2016

OPM:Can not find one alive process, Page cannot be displayed in 11i after running ADCFGCLONE

Issue :

 After running adcfgclone in apps Tier in 11i , It got completed without any error and When we tried to login the First page comes but when we try to click on homepage, it redirects to blank page which says "Page cannot be displayed" instead of login page.

In Apache Error log:
--------------------------

iAS Home/Apache/Apache/logs :

[Tue Apr  5 20:41:43 2016] [error] [client 10.1.163.122] File does not exist:
[Tue Apr  5 20:45:05 2016] [error] [client 10.1.163.122] File does not exist: /u01/app/doyen/doyencomn/html/US/ICXINDEX_doyen_usws1rdvcm04.htm
[Tue Apr  5 20:45:22 2016] [error] OPM:Can not find one alive process
[Tue Apr  5 20:45:22 2016] [error] [client 10.1.163.122] File does not exist: /servlets/weboam/oam/oamLogin/oam/oamLogin
[Tue Apr  5 20:51:13 2016] [error] OPM: EW: Fail to start process with mod=JServ and grp=DiscoGroup, it's possible that your configuration file is not correct.
[Tue Apr  5 20:51:13 2016] [error] OPM: EW: Fail to start process with mod=JServ and grp=OACoreGroup, it's possible that your configuration file is not correct.
[Tue Apr  5 20:51:13 2016] [error] OPM: EW: Fail to start process with mod=JServ and grp=OACoreGroup, it's possible that your configuration file is
[Tue Apr  5 21:12:18 2016] [notice] Oracle HTTP Server Powered by Apache/1.3.19 configured -- resuming normal operations
[Tue Apr  5 21:25:42 2016] [error] [client 10.1.163.122] File does not exist: /u01/app/doyen/doyencomn/portal/doyen_doyendb02/favicon.ico
[Tue Apr  5 21:25:45 2016] [error] [client 10.1.163.122] File does not exist: /u01/app/doyen/doyencomn/portal/doyen_doyendb02/favicon.ico
[Tue Apr  5 21:25:45 2016] [error] OPM:Can not find one alive process

In Jserv Logs:
----------------

[apdoyen@doyendb02 jvm]$ pwd
/u01/app/doyen/doyenora/iAS/Apache/Jserv/logs/jvm

vi OACoreGroup.4.stderr

Please use CMSClassUnloadingEnabled in place of CMSPermGenSweepingEnabled in the future
Could not create the Java virtual machine.

vi XmlSvcsGrp.0.stdout

/usr/bin/jdk1.6.0_30/jre/bin/java -DCLIENT_PROCESSID=8314 -verbose:gc -Xmx1600M -Xms1600M -XX:MaxPermSize=128M -XX:NewRatio=2 -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:
+PrintClassHistogram -XX:+UseTLAB -XX:+CMSPermGenSweepingEnabled -XX:+CMSClassUnloadingEnabled -Djava.awt.headless=false -DFND_TOP=/u01/app/doyen/doyenappl/fnd/11.5.0 -OA_HTML=/u01/app/doyen/doyencomn/html/ -DOA_MEDIA=/u01/app/doyen/doyencomn/java/oracle/apps/media/ -DOASMTPServer=doyendb02.hrbl.net -DWebProxyHost=doyendb02.hrbl.net -
DWebProxyPort=80 -DWebProxyByPassDomain=hrbl.net -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.ssl=false -
DOASSLCACertFile=/u01/app/doyen/doyenora/iAS/Apache/Apache/conf/ssl.crt/ca-bundle.crt -DOXTAInPoolSize=1 -DOXTAOutThreads=1 -DOXTAOutUseProxy=false -
DOXTAOutProxyHost=doyendb02.hrbl.net -DOXTAOutProxyPort=80 -DCOMMON_TOP=/u01/app/doyen/doyencomn -DOXTALogDebugMsg=false -DEXTERNAL_URL=http://doyendb02.hrbl.net:8016 -
Djbo.323.compatible=true -DAPPLRGF=/u01/app/doyen/doyencomn/rgf/doyen_doyendb02 -DJTFDBCFILE=/u01/app/doyen/doyenappl/fnd/11.5.0/secure/doyenapps01_doyen.dbc -
Dservice.Logging.common.filename=/u01/app/doyen/doyencomn/temp/ibe.log -Dframework.Logging.system.filename=/u01/app/doyen/doyencomn/temp/fwsys.log -
DIMT_COM_PROPERTY_FILE=/u01/app/doyen/doyenappl/imt/11.5.0/admin/scripts/imtjserv.properties -Dpoolsize=100 -Dminpoolsize=10 -Dpoolincrement=10 -Dpooldelayincrement=3 -
DBNEDBCFILE=/u01/app/doyen/doyenappl/fnd/11.5.0/secure/doyenapps01_doyen.dbc -Djserv.session.getValue.instrument=false org.apache.jserv.JServ -opmpropfile
/u01/app/doyen/doyenora/iAS/Apache/Jserv/etc/xmlsvcs.properties -opmhost doyendb02.hrbl.net -opmport 8116 -opmgrp XmlSvcsGrp -opmindex 0 -opmprocid 7
-------------------------------
Error occurred during initialization of VM
Could not reserve enough space for object heap
---------------------------------------------------

Solution  :
-----------
Cause :
---------

System was unable to reserve 1600M java heap memory for the jvm …

Solution : 

Reduced the java heap memory to be used by the jvm to 512M (updated the s_jvm_options  in the CONTEXT_FILE and ran autoconfig)


Before:
-verbose:gc –Xmx1600M –Xms1600M -XX:MaxPermSize=128M -XX:NewRatio=2 -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:
+PrintClassHistogram -XX:+UseTLAB -XX:+CMSClassUnloadingEnabled

After:

-verbose:gc -Xmx512M -Xms512M -XX:MaxPermSize=128M -XX:NewRatio=2 -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -
XX:+PrintClassHistogram -XX:+UseTLAB -XX:+CMSClassUnloadingEnabled

After this Run Autoconfig and retest the issue.

Reference : Error 'Could not reserve enough space for object heap' During Initialization of JVM (Doc ID 1028915.1)

Tuesday, March 29, 2016

Cause: The field PARAMETER.CONFIG could not be located or read in R12.2.4

Problem Summary
------------------------

In R12.2.4 After clone , the concurrent ouput/log file shows the error Cannot read value from field PARAMETER.CONFIG


Problem Description
-------------------------

After clone we get the following error in log and output file , but the request completed Normal


Error in the Output/log file :
---------------------------------

Cause: The field PARAMETER.CONFIG could not be located or read.

Action: This error is normally the result of an incorrectly-entered field name string in a trigger, or a field name string that does not uniquely specify a field in your form.

Correct your trigger logic to precisely specify a valid field.


Env Details:
--------------

2 DB node , 2 CM nodes , 2 APP nodes


To find out the issue :
-------------------------

Enable Trace for FNDFS in all the nodes (CM) in listener file :
--------------------------------------------------------------------------

( SID_DESC = ( SID_NAME = FNDFS  )
                 ( ORACLE_HOME = /u01/app/doyuat/fs1/EBSapps/10.1.2 )
                 ( PROGRAM = /u01/app/DOYuat/fs1/EBSapps/appl/fnd/12.0.0/bin/FNDFS )
                 (

envs='EPC_DISABLED=TRUE,NLS_LANG=American_America.UTF8,LD_LIBRARY_PATH=/u01/app/DOYuat/fs1/EBSapps/10.1.2/lib32:/u01/app/DOYuat/fs1/EBSapps/10.1.2/lib:/usr/X11R6/lib:/usr/openwin/lib:/u01/app/DOYuat/fs1/EBSapps/10.1.2/jdk/jre/lib/i386:/u01/app/DOYuat/fs1/EBSapps/10.1.2/jdk/jre/lib/i386/server:/u01/app/DOYuat/fs1/EBSapps/10.1.2/jdk/jre/lib/i386/native_threads:/u01/app/DOYuat/fs1/EBSapps/appl/sht/12.0.0/lib,SHLIB_PATH=/u01/app/DOYuat/fs1/EBSapps/10.1.2/lib32:/u01/app/DOYuat/fs1/EBSapps/10.1.2/lib:/usr/X11R6/lib:/usr/openwin/lib:/u01/app/DOYuat/fs1/EBSapps/10.1.2/jdk/jre/lib/i386:/u01/app/DOYuat/fs1/EBSapps/10.1.2/jdk/jre/lib/i386/server:/u01/app/DOYuat/fs1/EBSapps/10.1.2/jdk/jre/lib/i386/native_threads:/u01/app/DOYuat/fs1/EBSapps/appl/sht/12.0.0/lib,LIBPATH=/u01/app/DOYuat/fs1/EBSapps/10.1.2/lib32:/u01/app/DOYuat/fs1/EBSapps/10.1.2/lib:/usr/X11R6/lib:/usr/openwin/lib:/u01/app/DOYuat/fs1/EBSapps/10.1.2/jdk/jre/lib/i386:/u01/app/DOYuat/fs1/EBSapps/10.1.2/jdk/jre/lib/i386/server:/u01/app/DOYuat/fs1/EBSapps/10.1.2/jdk/jre/lib/i386/native_threads:/u01/app/DOYuat/fs1/EBSapps/appl/sht/12.0.0/lib,APPLFSTT=DOYUAT_806_BALANCE;DOYUATB_FO;DOYUATB;DOYUAT;DOYUATA;DOYUAT_FO;DOYUATA_FO,APPLFSWD=/u01/app/DOYuat/fs1/inst/apps/DOYUAT_doyensys01/appl/admin;/u01/app/DOYuat/fs1/inst/apps/DOYUAT_doyensys01/appltmp;/u01/app/DOYuat/fs1/FMW_Home/Oracle_EBS-app1/applications/oacore/html/oam/nonUix/launchMode/restricted,FNDFS_LOGGING=ON,FNDFS_LOGFILE=/u01/app/DOYuat/fs1/inst/apps/DOYUAT_doyensys01/logs/ora/10.1.2/network/FNDFS_DOYuat.log' )))


Line to be added in the listener for enabling trace and path of the log file :
---------------------------------------------------------------------------

FNDFS_LOGGING=ON,FNDFS_LOGFILE=/u01/app/doyuat/fs1/inst/apps/DOYUAT_doyensys01/logs/ora/10.1.2/network/FNDFS_doyuat.log'

Now bounce the listener and check for the request log/output files.

a) Retest the issue .

b) You will get the FNDFS log output in the specified location as mentioned in the listener (In the node in which the particular request try to fetch log/output file).

c) Now the error in the log file shows when connecting to the Node using the TWO_TASK it errored out.


Solution :
-----------

1. Check the TWO_TASK in Application nodes.(ex.It can be DOYUAT_BALANCE)
2. Check the listener file entry in Application Tier for the parameter APPLFSTT for which the TWO_TASK is included.
3. The same two_task entry should be included in both the CM nodes APPLFSTT value in listener file.
4. The Two_TASK value for CM nodes may be different like DOYUAT1 and DOYUAT2 for each nodes. But listener file should have included the Application's TWO_TASK for APPLFSTT.
5. The same value should be added in TNSNAMES file in both the CM NODES through which the Application node will connect to the CM node.
6. As a permenent fix change the value of APPLFSTT in XML file and run Autoconfig in CM node.


Following tns entry needs to be added in the tnsnames.ora in Both CM Nodes :
-------------------------------------------------------------------------------

DOYUAT_BALANCE=
        (DESCRIPTION=
            (ADDRESS_LIST=
                (LOAD_BALANCE=YES)
                (FAILOVER=YES)
                (ADDRESS=(PROTOCOL=tcp)(HOST=doyensys02.doyen.net)(PORT=1526))
                (ADDRESS=(PROTOCOL=tcp)(HOST=doyensys01.doyen.net)(PORT=1526))
            )
            (CONNECT_DATA=
                (SERVICE_NAME=DOYUAT)
            )
        )


After making the Entry in parameter (APPLFSTT)  in listener file in Both CM nodes  :


( SID_DESC = ( SID_NAME = FNDFS  )
                 ( ORACLE_HOME = /u01/app/doyuat/fs1/EBSapps/10.1.2 )
                 ( PROGRAM = /u01/app/DOYuat/fs1/EBSapps/appl/fnd/12.0.0/bin/FNDFS )
                 (envs='EPC_DISABLED=TRUE,NLS_LANG=American_America.UTF8,LD_LIBRARY_PATH=/u01/app/DOYuat/fs1/EBSapps/10.1.2/lib32:/u01/app/DOYuat/fs1/EBSapps/10.1.2/lib:/usr/X11R6/lib:/usr/openwin/lib:/u01/app/DOYuat/fs1/EBSapps/10.1.2/jdk/jre/lib/i386:/u01/app/DOYuat/fs1/EBSapps/10.1.2/jdk/jre/lib/i386/server:/u01/app/DOYuat/fs1/EBSapps/10.1.2/jdk/jre/lib/i386/native_threads:/u01/app/DOYuat/fs1/EBSapps/appl/sht/12.0.0/lib,SHLIB_PATH=/u01/app/DOYuat/fs1/EBSapps/10.1.2/lib32:/u01/app/DOYuat/fs1/EBSapps/10.1.2/lib:/usr/X11R6/lib:/usr/openwin/lib:/u01/app/DOYuat/fs1/EBSapps/10.1.2/jdk/jre/lib/i386:/u01/app/DOYuat/fs1/EBSapps/10.1.2/jdk/jre/lib/i386/server:/u01/app/DOYuat/fs1/EBSapps/10.1.2/jdk/jre/lib/i386/native_threads:/u01/app/DOYuat/fs1/EBSapps/appl/sht/12.0.0/lib,LIBPATH=/u01/app/DOYuat/fs1/EBSapps/10.1.2/lib32:/u01/app/DOYuat/fs1/EBSapps/10.1.2/lib:/usr/X11R6/lib:/usr/openwin/lib:/u01/app/DOYuat/fs1/EBSapps/10.1.2/jdk/jre/lib/i386:/u01/app/DOYuat/fs1/EBSapps/10.1.2/jdk/jre/lib/i386/server:/u01/app/DOYuat/fs1/EBSapps/10.1.2/jdk/jre/lib/i386/native_threads:/u01/app/DOYuat/fs1/EBSapps/appl/sht/12.0.0/lib,APPLFSTT=DOYUAT_BALANCE;DOYUAT_806_BALANCE;DOYUATB_FO;DOYUATB;DOYUAT;DOYUATA;DOYUAT_FO;DOYUATA_FO,APPLFSWD=/u01/app/DOYuat/fs1/inst/apps/DOYUAT_doyensys01/appl/admin;/u01/app/DOYuat/fs1/inst/apps/DOYUAT_doyensys01/appltmp;/u01/app/DOYuat/fs1/FMW_Home/Oracle_EBS-app1/applications/oacore/html/oam/nonUix/launchMode/restricted,FNDFS_LOGGING=ON,FNDFS_LOGFILE=/u01/app/DOYuat/fs1/inst/apps/DOYUAT_doyensys01/logs/ora/10.1.2/network/FNDFS_DOYuat.log' )   )  )


Monday, July 20, 2015

ORA-02085: database link DOYEN_REMOTEDB.DOYENSYS@USERNAME connects to DOYEN_REMOTEDB

ERROR:
=====

ORA-02085: database link DOYEN_REMOTEDB.DOYENSYS@USERNAME connects to DOYEN_REMOTEDB

SCENARIO:
========

SQL> select * from dual@DOYEN_REMOTEDB@USERNAME;
select * from dual@DOYEN_REMOTEDB@USERNAME

ERROR in líne 1:

ORA-02085: database link DOYEN_REMOTEDB.DOYENSYS@USERNAME connects to DOYEN_REMOTEDB

REASON :
=======

Global name in the database connecting to the remote is set to TRUE.

select name, value from v$parameter where name in ('db_name', 'db_domain', 'global_names');
NAME                 VALUE
-------------------- ----------------------------------------
db_domain
global_names         TRUE
db_name              mydb

SOLUTION :
=========

Simply at local db set

alter system set global_Names = FALSE;

Now it will work fine.

Wednesday, May 27, 2015

ORA-04031: unable to allocate bytes of shared memory ("","","","")


Problem Description:
===============

Archive log has been shipped from primary but not able to apply in standby.


Error in DB Alert log:
================

Completed Media Recovery
Checking to start in-flux buffer recovery from SCN 1504.3738248416 to SCN  (non-inclusive) 1504.3738248416

ORA-00283: recovery session canceled due to errors
ORA-01237: cannot extend datafile 113
ORA-01110: data file 113: '+DATA/doyensys_p/datafile/sysaux.408.874229253'
ORA-17505: ksfdrsz:1 Failed to resize file to size 1126400 blocks
ORA-00569: Failed to acquire global enqueue.
Slave exiting with ORA-283 exception
ORA-00283: recovery session canceled due to errors
ORA-01237: cannot extend datafile 113
ORA-01110: data file 113: '+DATA/doyensys_p/datafile/sysaux.408.874229253'
ORA-17505: ksfdrsz:1 Failed to resize file to size 1126400 blocks
ORA-00569: Failed to acquire global enqueue.


When we checked for the size, We have enough size in the Disk. But still it shows the above error. So we checked for the asm log.


Asm alert log :
===========

Wed May 20 19:55:29 2015
Errors in file /u01/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_ora_95265.trc  (incident=353797):
ORA-04031: unable to allocate  bytes of shared memory ("","","","")
Incident details in: /u01/app/oracle/diag/asm/+asm/+ASM1/incident/incdir_353797/+ASM1_ora_95265_i353797.trc
Wed May 20 19:55:30 2015

Solution:
========

There was some memory issue happened in the asm instance though it has the enough memory.So we have just bounced the asm instance.

1. Recovery needs to be cancelled, But it has already been done.
2. shut down the database
3. connect to asm instance
4. srvctl stop asm -n nodename
5. srvctl start asm -n nodename
6. Now start the database in mount and started the recovery.

Tuesday, December 2, 2014

ORA-28000 "the account is locked" in the standby database

Scenario:
=======

The user attempts to logon and gets an error ORA-28000 "the account is locked" in the primary database which is configure with Dataguard.  The database administrator unlock this user's account in the primary database.  Still, the user cannot connect to the standby, getting the same error ORA-28000 for the particular user, But the account status shows open in the standby.

In Standby :
=========

SYS@DOYEN > conn SARAVANA/saravana123123
ERROR:
ORA-28000: the account is locked

Warning: You are no longer connected to ORACLE.

SYS@DOYEN> select username,account_status from dba_users where username='SARAVANA';

USERNAME                                                                                   ACCOUNT_STATUS
------------------------------------------------------------------------------------------ 
SARAVANA                                                                                               OPEN


SYS@DOYEN > @check_db

                                                                                                                 Remote
       DbId Name      Log Mode     Open Mode            Database Role    Created             Resetlogs           Archive
----------- --------- ------------ -------------------- ---------------- ------------------- ------------------- --------
  603299473 DOYEN   ARCHIVELOG   READ ONLY WITH APPLY PHYSICAL STANDBY 11/16/2013 20:16:17 07/22/2014 03:54:21 ENABLED


Solution :
=======

 This is because the standby is open read-only and cannot update any tables.  When a user's account has to be locked on the standby database, it is locked only in memory there.

A privileged user (sysdba, the database administrator) must logon to the standby and unlock the account there.  A message ORA-28015 "Account unlocked, but the database is open for read-only access" confirms that the account is now unlocked in the standby database.  From then on, the user can logon to the standby database without getting any error.

SYS@DOYEN > ALTER USER SARAVANA ACCOUNT UNLOCK;
ALTER USER SARAVANA ACCOUNT UNLOCK
*
ERROR at line 1:
ORA-28015: Account unlocked, but the database is open for read-only access

SYS@DOYEN > conn SARAVANA/saravana123123
Connected.


Reference :
========

ORA-28000 "the account is locked" in the standby database, even after the account was unlocked in the primary. (Doc ID 1600401.1)

Friday, September 26, 2014

ERROR: NMO not setuid-root (Unix-only)

ERROR:

NMO not setuid-root (Unix-only).

2014-09-26 08:48:57,680 [14768:EDA3E961:GC.SysExecutor.23 (NmoHealthTask)] INFO - Job Attributes: {JobID=AgentHealthMonitor:14117357376
80, KeepAliveRequest=false} Job Command: {/u01/app/oracle/product/agent12c/sbin/nmo}
2014-09-26 08:48:57,714 [14768:EDA3E961] WARN - NmoStatus: Unable to verify nmo.
oracle.sysman.gcagent.target.interaction.execution.JobExecutionMgr$AuthenticatedJobExecutionException: ERROR: NMO not setuid-root (Unix-only)

The above error can result with the the below error:

EM Event: Warning:DOYEN.com - Agent has stopped monitoring. The following errors are reported : Read timed out.

Cause 1: 

The /root.sh script was not run after the Agent installation. So, the owner of the nmo, nmb and nmo executables is not the root user:

$ cd /sbin
$ ls -ltr nm*


For Example:

$ cd /u03/cloud12c/agent12c/sbin
$ ls -al nm*
-rwxr-xr-x 1 oracle oracle 22754 Feb 11 11:08 nmb
-rwxr-xr-x 1 oracle oracle 57835 Feb 11 11:10 nmhs
-rwxr-xr-x 1 oracle oracle 40478 Jan 31 16:57 nmo
...
The correct permissions should have been set by executing the file /root.sh as root at the Agent installation/deployment time.

Note :
=====
After running the root.sh the following file will be created. Do confirm the same.

/etc/oragchomelist


You should get:

$ ls -al nm*
-rwxr-xr-x  1 root oracle 585242 Jul  6  2013 nmosudo
-rws--x---  1 root oracle  40007 Jul  6  2013 nmo
-rws--x---  1 root oracle  57835 Jul  6  2013 nmhs
-rws--x---  1 root oracle  22754 Jul  6  2013 nmb
-rwxr-xr-x  1 root oracle  43494 Jul  6  2013 nmgsshe


Cause 2 :

 The root.sh script may have been run but the mount point for the is set with a 'nosuid' / 'nosetuid' option.

This can be verified by doing the mount command:

mount
or
mount | grep nosuid
and
mount | grep nosetuid
or
check the entries in files: /etc/vfstab, /etc/fstab

The 'nosetuid' option prevents mounted programs that have setuid permission to run with the permissions of their owners, regardless of who starts them. Without this

option, if a program with setuid permission is owned by root, it will run with root permissions, regardless of who starts it.
Sometimes 'nosuid' option is specified to protect the system against setuid programs that may run as root and damage the system.

SOLUTION :

Solution for Cause 1 :

Login to the Cloud Control Agent host as root.
Go to the Cloud Control Agent ORACLE_HOME directory

Example: /u01/app/oracle/product/Middleware/agent/core/12.1.0.1.0

Execute the root.sh script:

# ./root.sh
Log in to the Cloud Control Agent host as the Agent binaries owner (oracle for example)
Check that the permissions are now correct:

$ ls -ltr
....
-rwxr-xr-x  1 root em 585242 Jul  6  2013 nmosudo
-rws--x---  1 root em  40007 Jul  6  2013 nmo
-rws--x---  1 root em  57835 Jul  6  2013 nmhs
-rws--x---  1 root em  22754 Jul  6  2013 nmb
-rwxr-xr-x  1 root em  43494 Jul  6  2013 nmgsshe
Stop and restart the Cloud Control Agent:

Example
$ cd /u01/app/oracle/product/Middleware/agent/core/12.1.0.1.0/bin
$ ./emctl stop agent
$ ./emctl start agent


Solution for Cause 2

If the mount point has nosuid option set, this problem should be corrected by the system administrator of the Unix system:

Backup, edit and save the file /etc/vfstab or /etc/fstab for the mount point where the Agent has been installed and remove the 'nosuid' / 'nosetuid' option.
On AIX operating system edit thefile  /etc/filesystems.
(u01 in our example)
Stop and restart the Cloud Control Agent

$ cd $ORACLE_HOME/bin

Example

$ cd /u01/app/oracle/product/Middleware/agent/core/12.1.0.1.0/bin
$ ./emctl stop agent
$ ./emctl start agent


Reference :
========

Fix the Cloud Control Agent 12c error 'ERROR: NMO Not Setuid-root (Unix-only)' (Doc ID 146527)