Active Users concurrent program takes long time (never completes) after clone:

Version : 12.1.1

Scenario:
=======

Active Users concurrent program never completes after clone:

Recently we did a clone (R12). Everything worked as expected, except for Active Users Concurrent program which never completed when submitted. The status remains Running ? Normal for very long

This is how the issue was approached and resolved.:

1)We tried to run the Active User request with trace enabled,

but no trace file was generated.Hence we ran the report manually from the command prompt:

$INST_TOP/ora/10.1.2/bin/appsrwrun.sh userid=apps/ mode=character report=$FND_TOP/reports/US/FNDSCURS.rdf batch=yes destype=file desname=./ddr2.out desformat=$FND_TOP/reports/HPL pagesize=132x66 traceopts=trace_all tracefile=ddr2.trc tracemode=trace_replace 
(should generate trace in the same location)

This command produced the following error:

$ $INST_TOP/ora/10.1.2/bin/appsrwrun.sh userid=apps/ mode=character report=$FND_TOP/reports/US/FNDSCURS.rdf batch=yes destype=file desname=./ddr2.out desformat=$FND_TOP/reports/HPL pagesize=132x66 traceopts=trace_all tracefile=ddr2.trc tracemode=trace_replace
Environment:Environment java.lang.UnsatisfiedLinkError: Can't find library rwu  (librwu.a or .so) in sun.boot.library.path or java.library.path
sun.boot.library.path=/devapps/apps/tech_st/10.1.2/jdk/jre/bin
java.library.path=/devapps/apps/tech_st/10.1.2/jdk/jre/bin:/devapps/apps/tech_st/10.1.2/jdk/jre/bin/classic:/devapps/apps/tech_st/10.1.2/lib32:/devapps/apps/tech_st/10.1.2/lib:/usr/dt/lib:/devapps/apps/tech_st/10.1.2/jdk/jre/bin:/devapps/apps/tech_st/10.1.2/jdk/jre/bin/classic:/devapps/apps/apps_st/appl/cz/12.0.0/bin:/devapps/apps/tech_st/10.1.2/lib32:/devapps/apps/tech_st/10.1.2/lib:/usr/dt/lib:/devapps/apps/tech_st/10.1.2/jdk/jre/bin:/devapps/apps/tech_st/10.1.2/jdk/jre/bin/classic:/devapps/apps/apps_st/appl/sht/12.0.0/lib:/devapps/apps/tech_st/10.1.2/jdk/jre/bin/sovvm:/usr/lib
Environment:getEnvironmentStrings java.lang.UnsatisfiedLinkError: Can't find library rwu  (librwu.a or .so) in sun.boot.library.path or java.library.path
sun.boot.library.path=/devapps/apps/tech_st/10.1.2/jdk/jre/bin
java.library.path=/devapps/apps/tech_st/10.1.2/jdk/jre/bin:/devapps/apps/tech_st/10.1.2/jdk/jre/bin/classic:/devapps/apps/tech_st/10.1.2/lib32:/devapps/apps/tech_st/10.1.2/lib:/usr/dt/lib:/devapps/apps/tech_st/10.1.2/jdk/jre/bin:/devapps/apps/tech_st/10.1.2/jdk/jre/bin/classic:/devapps/apps/apps_st/appl/cz/12.0.0/bin:/devapps/apps/tech_st/10.1.2/lib32:/devapps/apps/tech_st/10.1.2/lib:/usr/dt/lib:/devapps/apps/tech_st/10.1.2/jdk/jre/bin:/devapps/apps/tech_st/10.1.2/jdk/jre/bin/classic:/devapps/apps/apps_st/appl/sht/12.0.0/lib:/devapps/apps/tech_st/10.1.2/jdk/jre/bin/sovvm:/usr/lib
REP-50125: Caught exception: java.lang.reflect.InvocationTargetException

2)Next we checked for the reports servers running by

$ rwdiag.sh -findAll

(This command also produced the following error)

Environment:Environment java.lang.UnsatisfiedLinkError: Can't find library rwu  (librwu.a or .so) in sun.boot.library.path or java.library.path
sun.boot.library.path=/devapps/apps/tech_st/10.1.2/jdk/jre/bin
java.library.path=/devapps/apps/tech_st/10.1.2/jdk/bin:/devapps/apps/tech_st/10.1.2/jdk/jre/bin/classic:/devapps/apps/tech_st/10.1.2/jdk/jre/bin:/devapps/apps/tech_st/10.1.2/jdk/jre/bin:/devapps/apps/tech_st/10.1.2/jdk/jre/bin/classic:/devapps/apps/tech_st/10.1.2/lib32:/devapps/apps/tech_st/10.1.2/lib:/usr/dt/lib:/devapps/apps/tech_st/10.1.2/jdk/jre/bin:/devapps/apps/tech_st/10.1.2/jdk/jre/bin/classic:/devapps/apps/apps_st/appl/cz/12.0.0/bin:/devapps/apps/tech_st/10.1.2/lib32:/devapps/apps/tech_st/10.1.2/lib:/usr/dt/lib:/devapps/apps/tech_st/10.1.2/jdk/jre/bin:/devapps/apps/tech_st/10.1.2/jdk/jre/bin/classic:/devapps/apps/apps_st/appl/sht/12.0.0/lib:/devapps/apps/tech_st/10.1.2/jdk/jre/bin/sovvm:/usr/lib
Environment:getEnvironmentStrings java.lang.UnsatisfiedLinkError: Can't find library rwu  (librwu.a or .so) in sun.boot.library.path or java.library.path
sun.boot.library.path=/devapps/apps/tech_st/10.1.2/jdk/jre/bin
java.library.path=/devapps/apps/tech_st/10.1.2/jdk/bin:/devapps/apps/tech_st/10.1.2/jdk/jre/bin/classic:/devapps/apps/tech_st/10.1.2/jdk/jre/bin:/devapps/apps/tech_st/10.1.2/jdk/jre/bin:/devapps/apps/tech_st/10.1.2/jdk/jre/bin/classic:/devapps/apps/tech_st/10.1.2/lib32:/devapps/apps/tech_st/10.1.2/lib:/usr/dt/lib:/devapps/apps/tech_st/10.1.2/jdk/jre/bin:/devapps/apps/tech_st/10.1.2/jdk/jre/bin/classic:/devapps/apps/apps_st/appl/cz/12.0.0/bin:/devapps/apps/tech_st/10.1.2/lib32:/devapps/apps/tech_st/10.1.2/lib:/usr/dt/lib:/devapps/apps/tech_st/10.1.2/jdk/jre/bin:/devapps/apps/tech_st/10.1.2/jdk/jre/bin/classic:/devapps/apps/apps_st/appl/sht/12.0.0/lib:/devapps/apps/tech_st/10.1.2/jdk/jre/bin/sovvm:/usr/lib
java.lang.reflect.InvocationTargetException
oracle.reports.RWException: IDL:oracle/reports/RWException:1.0

This indicated that the reports executables need to be relinked as per the note Concurrent Reports UnsatisfiedLinkError [ID 849369.1]

Solution:
======

To implement the solution, we executed the following steps:

1. Relink the reports executables:

cd $ORACLE_HOME/reports/lib32
make -f ins_reports.mk install

(Error again!!!!)
============

Error while trying to relink :  Even though application tier was down we got the error:
      /devapps/apps/tech_st/10.1.2/bin/mkexp.sh /devapps/apps/tech_st/10.1.2/lib32//librwu.a  > librwuso.exp
        rm -f rwsutil.o rwspid.o ;  /bin/ar x /devapps/apps/tech_st/10.1.2/lib32//librwu.a  rwsutil.o rwspid.o ;  (LD_OPTIONS="-zmultidefs";  ld -o /devapps/apps/tech_st/10.1.2/lib32//librwu.so -bM:SRE -bnoentry -bnolibpath  -bE:librwuso.exp rwsutil.o rwspid.o  -lm -lc -lm  -ldl /devapps/apps/tech_st/10.1.2/lib32/libclntsh.a -lpthreads  -L/usr/lpp/X11/lib/R6 -L/usr/lpp/X11/lib/R6/Motif2.1 -lXm -lXt -lX11 -lm `cat /devapps/apps/tech_st/10.1.2/lib32/sysliblist` /devapps/apps/tech_st/10.1.2/lib32/libclntst10.a )
        rm -f zrhitem.o zrhcursor.o;  /bin/ar x /devapps/apps/tech_st/10.1.2/lib32//librw.a zrhitem.o zrhcursor.o
        /devapps/apps/tech_st/10.1.2/bin/mkexp.sh /devapps/apps/tech_st/10.1.2/lib32/libmmcm.a > temp1.exp
        /devapps/apps/tech_st/10.1.2/bin/mkexp.sh /devapps/apps/tech_st/10.1.2/lib32/librw.a > temp2.exp
        cat temp1.exp temp2.exp > temp.exp
        sort -u temp.exp > librw.exp
        rm temp1.exp temp2.exp temp.exp
ld: 0711-851 SEVERE ERROR: Output file: /devapps/apps/tech_st/10.1.2/lib32//librw.so
        The file is in use and cannot be overwritten.
make: 1254-004 The error code from the last command is 12.

This meant that the file was in use :

We referred to the following Metalink note ID

What are the steps needed to Clean up the Libraries during Relinking of Reports on AIX? [ID 834577.1]

In order to overcome this issue:

su - root

Issued the following command:

/usr/sbin/slibclean

Then login as a normal user and now relink the Reports Services.

$ORACLE_HOME/reports/lib32> make -f ins_reports.mk install

2. Retest for the issue
Success!!!
Active user completed Normal