An error occurred while attempting to establish an Applications File Server connection with the node FNDFS_Hostname.

 

Error:

"An error occurred while attempting to establish an Applications File Server connection with the node FNDFS_<hostname>. There may be a network configuration problem, or the TNS listener on node FNDFS_<hostname> may not be running. Please contact your system administrator."

Environment:

1) 19c database and 12.2.9 environment , Three node database and 2 Node (PCP)

Recent Changes

Ran Auto config in all db and application node as a part of patching.

Issue:

1) After starting all database and application services,

2) We are not able to see output and logs from the front end.

3) All application services started without error.

Solution:

As the TNS and Context file has been backed up already before patching, We restore the whole TNS with the permission 755 and then the issue gets resolved.

Scenario:

Encountered the following error in Apps Front end while trying to access concurrent program logs/Output files which are generated before switch over.

Error:
An error occurred while attempting to establish an Applications File Server connection with the node FNDFS_STBYAPP2. There may be a network configuration problem, or the TNS listener on node FNDFS_STBYAPP2 may not be running. Please contact your system administrator.

Reason:

The log and output file is pointing to the DR Apps Node, As it was generated during the DR was Primary.  But the current log and ouput files has no issues in viewing.

Solution:

Update the node name for Log and Output files with current App server Node Name

select count(*) from fnd_concurrent_requests where LOGFILE_NODE_NAME  = 'STBYAPP1';
update fnd_concurrent_requests set LOGFILE_NODE_NAME = 'PRIMAPP1' where LOGFILE_NODE_NAME  = 'STBYAPP1';

select count(*) from fnd_concurrent_requests where LOGFILE_NODE_NAME  = 'STBYAPP2';
update fnd_concurrent_requests set LOGFILE_NODE_NAME = 'PRIMAPP2' where LOGFILE_NODE_NAME = 'STBYAPP2';

select count(*) from fnd_concurrent_requests where OUTFILE_NODE_NAME = 'STBYAPP1';
update fnd_concurrent_requests set OUTFILE_NODE_NAME = 'PRIMAPP1' where OUTFILE_NODE_NAME = 'STBYAPP1';

select count(*) from fnd_concurrent_requests where OUTFILE_NODE_NAME = 'STBYAPP2';
update fnd_concurrent_requests set OUTFILE_NODE_NAME = 'PRIMAPP2' where OUTFILE_NODE_NAME = 'STBYAPP2';

Oracle EBS Forms Not opening in IE 11

Env Details :

Oracle EBS version 12.1.3
Windows  7 Prof SP1
Java installed 8.151 - 32 bit  installed in the path C:\Program Files (x86)\Java\jre1.8.0_151\bin\javaw.exe
IE - 11 

Solution :

Needs to check the following if your forms did not open and asks every time to install the 7 u65 version of jre.

1. Install the 32 bit 8.151 version of JRE though you the 64 bit version of windows 7.
2. While installing , Right click -> Run as a Administrator.
3. Give necessary permission for java folder to the users as well to run and execute - If it is not there.
4. Go to C:\Program Files (x86)\Java\jre1.8.0_151\bin -> Right click on Java, Javacpl, javaws  -> Compatibility -> Run this Program in Comatibilty mode and select Win 7 asnd check privilege level as Run as Administrator. And change setting for all users as well.
5. Control Panel -> Java -> General -> Settings (under Temporary Files Settings)  "Delete Files".
6. In the internet explorer 11 , navigate to settings –> Compatibility View Settings . Then add Domain name in the URL(ex: saravana.com) using Add this website option.
7. Navigate to settings –> Internet options –> Security -> internet -> and Change the Custom level to minimum and change the option Enable XSS filter to  Disable and Use Pop-up Blocker Disabled. Also make the ActiveX controls to enabled.
8. Now Try to open the forms.

ORA-04063: rule set "APPLSYS.WF_JAVA_DEFERRED_R" has errors

Scenario:
------------

Got the Forms error while adding responsibility to a user in R12.2.5. So, I did the workflow sync validation and it does not fixed. So, I ran autoconfig in Apps tier and it end up with Error and ran in db node and that too endup with the same error.

1. Auto config in apps Node completed with the following Error:

WARNING: [AutoConfig Error Report]
The following report lists errors AutoConfig encountered during each
phase of its execution.  Errors are grouped by directory and phase.
The report format is:
         

  [PROFILE PHASE]
  AutoConfig could not successfully execute the following scripts:
    Directory: /u01/app/DOYEN/fs2/FMW_Home/webtier/perl/bin/perl -I /u01/app/DOYEN/fs2/FMW_Home/webtier/perl/lib/5.10.0 -I /u01/app/DOYEN/fs2/FMW_Home/webtier/perl/lib/site_perl/5.10.0 -I /u01/app/DOYEN/fs2/EBSapps/appl/au/12.0.0/perl -I /u01/app/DOYEN/fs2/FMW_Home/webtier/ohs/mod_perl/lib/site_perl/5.10.0/x86_64-linux-thread-multi /u01/app/DOYEN/fs2/inst/apps/DOYEN_sysapp02/admin/scripts/adexecsql.pl sqlfile=/u01/app/DOYEN/fs2/inst/apps/DOYEN_sysapp02/admin/install
      afwebprf.sql            INSTE8_PRF         1

AutoConfig is exiting with status 1

2. Autoconfig in the DB node completed with the following error:

WARNING: [AutoConfig Error Report]
The following report lists errors AutoConfig encountered during each
phase of its execution.  Errors are grouped by directory and phase.
The report format is:
         

  [PROFILE PHASE]
  AutoConfig could not successfully execute the following scripts:
    Directory: /u01/orahome/u01db/11.2.0.4/appsutil/install/DOYEN_system01
      afdbprf.sh              INSTE8_PRF         1

AutoConfig is exiting with status 1


Now when i tried running the script (afwebprf.sql) manually which errored out in apps autoconfig and i faced the same error which we got in forms.

Now i tried running the script (afdbprf.sh) manually which errored out in db node and found the same error:

[orau01@system01:DOYEN_system01]# sh afdbprf.sh

afdbprf.sh started at Thu Jun 22 19:54:29 IDT 2017

The environment settings are as follows ...

       ORACLE_HOME : /u01/orahome/u01db/11.2.0.4
        ORACLE_SID : DOYEN
              PATH : /u01/orahome/u01db/11.2.0.4/perl/bin:/u01/orahome/u01db/11.2.0.4/perl/bin:/u01/orahome/u01db/11.2.0.4/bin:/usr/bin:/usr/sbin:/u01/orahome/u01db/11.2.0.4/appsutil/jre/bin:/bin:/usr/bin/X11:/usr/local/bin:/usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin:.:.
      Library Path : /u01/orahome/u01db/11.2.0.4/lib:/usr/X11R6/lib:/usr/openwin/lib:/u01/orahome/u01db/11.2.0.4/lib:/usr/dt/lib:/u01/orahome/u01db/11.2.0.4/ctx/lib

Executable : /u01/orahome/u01db/11.2.0.4/bin/sqlplus

Enter the APPS username: apps
Enter the APPS password:
SQL*Plus: Release 11.2.0.4.0 Production on Thu Jun 22 19:54:33 2017
Copyright (c) 1982, 2013, Oracle.  All rights reserved.

Enter value for 1: Enter value for 2: Enter value for 3: Connected.
[ FND_DB_WALLET_DIR ]
Application Id : 0
Profile Value  : /u01/orahome/u01db/11.2.0.4/appsutil/wallet
Level Name: SITE
INFO       : Updated/created profile option value.
.
[ ECX_UTL_XSLT_DIR ]
Application Id : 174
Profile Value  : /usr/tmp
Level Name: SITE
INFO       : Updated/created profile option value.
.
[ ECX_UTL_LOG_DIR ]
Application Id : 174
Profile Value  : /usr/tmp
Level Name: SITE
INFO       : Updated/created profile option value.
.
[ BIS_DEBUG_LOG_DIRECTORY ]
Application Id : 0
Profile Value  : /usr/tmp
Level Name: SITE
INFO       : Updated/created profile option value.
.
[ UTL_FILE_LOG ]
Application Id : 1
Profile Value  : /usr/tmp, /u01/logout/tmp, /tmp/, /u01/oratmp/tempdata,
/u01/logout/Agile, /u01/logout/Agile/in, /u01/logout/Agile/out,
/u01/winshare/APAC/AP, /u01/winshare/crm/scorecard,
/u01/winshare/APAC/WESTPAC, /u01/logout/cm,
/u01/orahome/u01db/11.2.0.4/appsutil/outbound/DOYEN_system01
Level Name  : SITE
INFO       : Error updating/creating profile option value.
.
begin
*
ERROR at line 1:
ORA-04063: rule set "APPLSYS.WF_JAVA_DEFERRED_R" has errors
ORA-06512: at "APPS.WF_EVENT", line 1194
ORA-06512: at "APPS.FND_WF_ENGINE", line 1025
ORA-06512: at "APPS.FND_PROFILE", line 1102
ORA-06512: at "APPS.FND_PROFILE", line 2353
ORA-06512: at "APPS.ADX_PRF_PKG", line 192
ORA-06512: at line 42

The above error seems to be an issue with the WF Ruleset.

Solution:
-----------

Checked the following Rule Set is valid and found to be invalid.

select object_name, object_type, status from dba_objects where object_name='WF_JAVA_DEFERRED_R';

WF_JAVA_DEFERRED_R RULE SET INVALID


This is because of the issues with the Workflow queues. So i did the clean up and enque processes as suggested in the below Note id.

After following the following noteid , It fixed the  issue:

Fixing Invalid Workflow Rule Sets such as WF_DEFERRED_R and Related Errors on Workflow Queues:ORA-24033 (Doc ID 337294.1)

Once it is done, I have checked the status of the Ruleset and it is valid.

select object_name, object_type, status from dba_objects where object_name='WF_JAVA_DEFERRED_R';

WF_JAVA_DEFERRED_R RULE SET VALID


Now , Ran the Autoconfig and it has completed Normal.

Cause: cleanup_node failed due to ORA-01427

Scenario:
------------

The concurrent requests are going to Error in R12.2.5 and cannot view the log files as well.


Error:
------

Cause: cleanup_node failed due to ORA-01427: single-row subquery returns more than one row
ORA-06512: at "APPS.FND_CP_FNDSM", line 91
ORA-06512: at line 1.

The SQL statement being executed at the time of the error was: begin fnd_cp_fndsm.mark_killed_fndsm(:nn); end;  and was executed from the file &ERRFILE.
Routine AFPEIM encountered an error while starting concurrent manager STANDARD with library /test/app/fs2/EBSapps/appl/fnd/12.0.0/bin/FNDLIBR.

Check that your system has enough resources to start a concurrent manager process. Contact your system : 13-JUN-2017 22:28:18
Starting STANDARD Concurrent Manager               : 13-JUN-2017 22:28:18

Could not initialize the Service Manager FNDSM_DOYSER_DEV. Verify that DOYSER has been registered for concurrent processing.
An error occured in client-side routine afpsmcsm for Service Manager FNDSM_DOYSER_DEV.  The routine returned code 126.

Check for preceding errors and as well as the service manager log file for further details."

ORACLE error 1427 in cleanup_node


Cause:
---------

After clone the node is not cleaned up which has the old values.

Solution:
----------

1. Shutdown the apps node.
2. login to db node and run the following.

sqlplus apps/apps

sql>exec fnd_conc_clone.setup_clean;

commit;

sql>@cmclean.sql

3. Run autoconfig in db tier.
4. Run autoconfig in the apps tier and start the services.

This should be fixed.

ORA-20100 : Temporary file creation for FND_FILE failed.

Scenario:
---------

Request completed with the following error:

ORA-20100 : Temporary file creation for FND_FILE failed.

Log file shows the following :
------------------------------------
+---------------------------------------------------------------------------+
Application Object Library: Version : 12.2

Copyright (c) 1998, 2013, Oracle and/or its affiliates. All rights reserved.

FNDOAMCOL: OAM Applications Dashboard Collection
+---------------------------------------------------------------------------+

Updated system property java.protocol.handler.pkgs to HTTPClient
Refreshing All ...
Refreshing Applications System Status ...
Refreshing Activity ...
Refreshing Configuration Changes ...
Refreshing Throughput ...
Refreshing Exceptions Summary ...
Refreshing User Alert Summary ...
Collecting metric values for each aplication tier node ...
Collecting metric values for each database instance ...
Done Refreshing All ...
Checking collected data for alerting ...
Checking long running requests for alerting ...
SQLException:
java.sql.SQLException: ORA-20100: ORA-20100: Temporary file creation for FND_FILE failed.
Directory &FILE_DIR is invalid, contact your system administrator.

Action: Check that the directory is a valid directory with write permissions on the database server node and is listed under utl_file_dir para
ORA-06512: at "APPS.FND_FILE", line 319
ORA-06512: at "APPS.FND_FILE", line 364
ORA-06512: at "APPS.FND_FILE", line 421
ORA-06512: at "APPS.FND_OAM_COLLECTION", line 2786
ORA-06512: at line 1

Oracle error : An unexpected system error has been detected in Application code. Please contact System Administrator for assistance.Successfully resubmitted concurrent program FNDOAMCOL with request ID 84242340 to start at 31-MAY-2017 23:45:52 (ROUTINE=AFPSRS)

+---------------------------------------------------------------------------+
No completion options were requested.

Output file size:
----------------------------------------------------------------------------+

Reason:
---------

The APPLPTEMP location points to a directory which is not in the utl_file directory in the database. So when the application user is trying to create a temp file in the location it is not allowed.


grep -i APPLPTMP $CONTEXT_FILE
         /doyen/app/DOYENSYS/fs2/inst/apps/DOYENSYS_sara1/temp


when i check the utl_file, It shows the following :

/usr/tmp


 Solution:
=======

1. Either add the following location in utl_file dir and restart the db /doyen/app/DOYENSYS/fs2/inst/apps/DOYENSYS_sara1/temp    or
2. Change the s_applptmp value to /usr/tmp and run autoconfig and try running the report again.

Configure Patch fs in R12.2

Scenario:

Assuming the Run and Patch fs are going to be in single node and the Run fs is already configured and the instance is up and running.

Steps to configure the patch File system in R12.2

Now to configure the patch fs.  Do the following

1.   Run the preclone in the Run fs.

[appdoyen@doyserv1 scripts]$ pwd

/test/app/TEST/fs2/inst/apps/TEST_doyserv1/admin/scripts

[appdoyen@doyserv1 scripts]$ adpreclone.pl appsTier

Copyright (c) 2011, 2014 Oracle Corporation

                        Redwood Shores, California, USA

                        Oracle E-Business Suite Rapid Clone

                                 Version 12.2

                      adpreclone Version 120.31.12020000.20

Enter the APPS User Password:

Enter the Weblogic AdminServer password :

 Checking the status of the Oracle WebLogic Administration Server....

 Running perl /test/app/TEST/fs2/EBSapps/appl/ad/12.0.0/patch/115/bin/adProvisionEBS.pl ebs-get-serverstatus -contextfile=/test/app/TEST/fs2/inst/apps/TEST_doyserv1/appl/admin/TEST_doyserv1.xml -servername=AdminServer -promptmsg=hide

The Oracle WebLogic Administration Server is up.

Running:

perl /test/app/TEST/fs2/EBSapps/appl/ad/12.0.0/bin/adclone.pl java=/test/app/TEST/fs2/EBSapps/comn/util/jdk64 mode=stage stage=/test/app/TEST/fs2/EBSapps/comn/clone component=appsTier method= appctx=/test/app/TEST/fs2/inst/apps/TEST_doyserv1/appl/admin/TEST_doyserv1.xml showProgress

 Setting the wls environment

Beginning application tier Stage - Mon May 22 18:39:00 2017

/test/app/TEST/fs2/EBSapps/comn/util/jdk64/bin/java -Xmx600M -DCONTEXT_VALIDATED=false -Doracle.installer.oui_loc=/oui -classpath /test/app/TEST/fs2/FMW_Home/webtier/lib/xmlparserv2.jar:/test/app/TEST/fs2/FMW_Home/webtier/jdbc/lib/ojdbc6.jar:/test/app/TEST/fs2/EBSapps/comn/java/classes:/test/app/TEST/fs2/FMW_Home/webtier/oui/jlib/OraInstaller.jar:/test/app/TEST/fs2/FMW_Home/webtier/oui/jlib/ewt3.jar:/test/app/TEST/fs2/FMW_Home/webtier/oui/jlib/share.jar:/test/app/TEST/fs2/FMW_Home/webtier/../Oracle_EBS-app1/oui/jlib/srvm.jar:/test/app/TEST/fs2/FMW_Home/webtier/jlib/ojmisc.jar:/test/app/TEST/fs2/FMW_Home/wlserver_10.3/server/lib/weblogic.jar:/test/app/TEST/fs2/FMW_Home/oracle_common/jlib/obfuscatepassword.jar  oracle.apps.ad.clone.StageAppsTier -e /test/app/TEST/fs2/inst/apps/TEST_doyserv1/appl/admin/TEST_doyserv1.xml -stage /test/app/TEST/fs2/EBSapps/comn/clone -tmp /tmp -method CUSTOM   -showProgress -nopromptmsg

Log file located at /test/app/TEST/fs2/inst/apps/TEST_doyserv1/admin/log/clone/StageAppsTier_05221839.log

  /     20% completed

Completed Stage...

2.     Now create a empty patch fs directory in our case  (fs1).

/test/app/TEST/fs1

3.    Now shut down the runfs (fs2) and copy the EBS directory from run to patch fs

Note:  Now you can startall services in the run fs.

[appdoyen@doyserv1 scripts]$ cd $RUN_BASE

[appdoyen@doyserv1 fs2]$ pwd

/test/app/TEST/fs2

[appdoyen@doyserv1 fs2]$ cp -rp EBSapps /test/app/TEST/fs1/

4.     Unset the env if anything is set already.

5.      Go inside /test/app/TEST/fs1/EBSapps/comn/clone/bin of Patch fs ie) fs1 here , and run adcfgclone.pl

6.      Run perl adcfgclone.pl appsTier

7.      Give the port pool incremented by one on the run fs. for ex. If run fs is running in port pool 36 you can give as 37. Also check if it is already allcated by using 

netstat -an|grep -i 8037

    

[appdoyen@doyserv1 bin]$ perl adcfgclone.pl appsTier

                     Copyright (c) 2002, 2015 Oracle Corporation

                        Redwood Shores, California, USA

                        Oracle E-Business Suite Rapid Clone

                                 Version 12.2

                      adcfgclone Version 120.63.12020000.56

              ***********************************************************

                            In AD-TXK Delta 7, we recommend you clone the run and patch

                            file systems in a single operation using the 'dualfs' option.

                            Separate cloning of the run and patch file systems will be deprecated

              ************************************************************

Enter the APPS password :

Enter the Weblogic AdminServer password :

Do you want to add a node (yes/no) [no] :

 Running: Context clone...

 Log file located at /test/app/TEST/fs1/EBSapps/comn/clone/bin/CloneContext_0522221213.log

Target System File Edition type [run] : patch

Enter the full path of Run File System Context file : /test/app/TEST/fs2/inst/apps/TEST_doyserv1/appl/admin/TEST_doyserv1.xml

Provide the values required for creation of the new APPL_TOP Context file.

Target System Fusion Middleware Home set to /test/app/TEST/fs1/FMW_Home

Target System Web Oracle Home set to /test/app/TEST/fs1/FMW_Home/webtier

Target System Appl TOP set to /test/app/TEST/fs1/EBSapps/appl

Target System COMMON TOP set to /test/app/TEST/fs1/EBSapps/comn

Target System Instance Top set to /test/app/TEST/fs1/inst/apps/TEST_doyserv1

Target System Port Pool [0-99] : 37

Checking the port pool 37

done: Port Pool 37 is free

Report file located at /test/app/TEST/fs1/inst/apps/TEST_doyserv1/admin/out/portpool.lst

The new APPL_TOP context file has been created :

  /test/app/TEST/fs1/inst/apps/TEST_doyserv1/appl/admin/TEST_doyserv1.xml

Check Clone Context logfile /test/app/TEST/fs1/EBSapps/comn/clone/bin/CloneContext_05348.log for details.

Running Rapid Clone with command:

Running:

perl /test/app/TEST/fs1/EBSapps/comn/clone/bin/adclone.pl java=/test/app/TEST/fs1/EBSapps/comn/clone/bin/../jre mode=apply stage=/test/app/TEST/fs1/EBSapps/comn/clone component=appsTier method=CUSTOM appctxtg=/test/app/TEST/fs1/inst/apps/TEST_doyserv1/appl/admin/TEST_doyserv1.xml showProgress contextValidated=true

  

FMW Pre-requisite check log file location : /test/app/TEST/fs1/EBSapps/comn/clone/FMW/logs/prereqcheck.log

Running: /test/app/TEST/fs1/EBSapps/comn/clone/FMW/t2pjdk/bin/java -classpath /test/app/TEST/fs1/EBSapps/comn/clone/prereq/webtier/Scripts/ext/jlib/engine.jar:/test/app/TEST/fs1/EBSapps/comn/clone/prereq/webtier/oui/jlib/OraPrereq.jar:/test/app/TEST/fs1/EBSapps/comn/clone/prereq/webtier/oui/jlib/OraPrereqChecks.jar:/test/app/TEST/fs1/EBSapps/comn/clone/prereq/webtier/oui/jlib/OraInstaller.jar:/test/app/TEST/fs1/EBSapps/comn/clone/prereq/webtier/oui/jlib/OraInstallerNet.jar:/test/app/TEST/fs1/EBSapps/comn/clone/prereq/webtier/oui/jlib/srvm.jar:/test/app/TEST/fs1/EBSapps/comn/clone/prereq/webtier/Scripts/ext/jlib/ojdl.jar:/test/app/TEST/fs1/EBSapps/comn/clone/prereq/webtier/Scripts/ext/jlib/ojdl2.jar:/test/app/TEST/fs1/EBSapps/comn/clone/prereq/webtier/Scripts/ext/jlib/ojdl-log4j.jar:/test/app/TEST/fs1/EBSapps/comn/clone/prereq/webtier/oui/jlib/xmlparserv2.jar:/test/app/TEST/fs1/EBSapps/comn/clone/prereq/webtier/oui/jlib/share.jar:/test/app/TEST/fs1/EBSapps/comn/clone/jlib/java oracle.apps.ad.clone.util.FMWOracleHomePreReqCheck -prereqCheckFMW -e /test/app/TEST/fs1/inst/apps/TEST_doyserv1/appl/admin/TEST_doyserv1.xml -stage /test/app/TEST/fs1/EBSapps/comn/clone -log /test/app/TEST/fs1/EBSapps/comn/clone/FMW/logs/prereqcheck.log

Beginning application tier Apply - Mon Feb 22 22:20:59 2016

/test/app/TEST/fs1/EBSapps/comn/clone/bin/../jre/bin/java -Xmx600M -DCONTEXT_VALIDATED=true -Doracle.installer.oui_loc=/oui -classpath /test/app/TEST/fs1/EBSapps/comn/clone/jlib/xmlparserv2.jar:/test/app/TEST/fs1/EBSapps/comn/clone/jlib/ojdbc6.jar:/test/app/TEST/fs1/EBSapps/comn/clone/jlib/java:/test/app/TEST/fs1/EBSapps/comn/clone/jlib/oui/OraInstaller.jar:/test/app/TEST/fs1/EBSapps/comn/clone/jlib/oui/ewt3.jar:/test/app/TEST/fs1/EBSapps/comn/clone/jlib/oui/share.jar:/test/app/TEST/fs1/FMW_Home/webtier/../Oracle_EBS-app1/oui/jlib/srvm.jar:/test/app/TEST/fs1/EBSapps/comn/clone/jlib/ojmisc.jar:/test/app/TEST/fs1/FMW_Home/wlserver_10.3/server/lib/weblogic.jar:/test/app/TEST/fs1/EBSapps/comn/clone/jlib/obfuscatepassword.jar  oracle.apps.ad.clone.ApplyAppsTier -e /test/app/TEST/fs1/inst/apps/TEST_doyserv1/appl/admin/TEST_doyserv1.xml -stage /test/app/TEST/fs1/EBSapps/comn/clone    -showProgress -nopromptmsg

Log file located at /test/app/TEST/fs1/inst/apps/TEST_doyserv1/admin/log/clone/ApplyAppsTier_05222221.log

  \    100% completed

Completed Apply...

Mon Feb 22 23:10:14 2016

Looking for incomplete CLONE record in ad_adop_session_patches table

The CLONE record status is COMPLETED

8.      Now source the env and startall .