Application restart was giving timeout error for http, oacore and oafm services.

        ERROR : Timed out( 100000 ): Interrupted Exception
        You are running adoafmctl.sh version 120.6.12000000.2
        Starting OPMN managed OAFM OC4J instance  ...

• Ran autoconfig on all node starting from db , ap and cm to sink all xml and config files of application. 

• Increase parameter in $ORA_CONFIG_HOME/10.1.3/opmn/conf/opmn.xml  file to fix timeout error in oacore and oafm.

    <start timeout="600" retry="2"/>    changed to <start timeout="1800" retry="2"/>

• Post timeout parameter change error is “adoacorectl.sh: exiting with status 150”  and  “adoafmctl.sh: exiting with status 150  “

Part of log.....

Program : /u001/SACH/apps/apps_st/appl/fnd/12.0.0/patch/115/bin/txkChkFormsDeployment.pl completed @ Thu Jul 21 05:45:42 2016

Perl script txkChkFormsDeployment.pl got executed successfully

adformsctl.sh: exiting with status 150

adformsctl.sh: check the logfile /u001/SACH/inst/apps/SACH_hqidlsachap01/logs/appl/admin/log/adformsctl.txt for more information ...


.end std out.
*** ALL THE FOLLOWING FILES ARE REQUIRED FOR RESOLVING RUNTIME ERRORS
*** Log File = /u001/SACH/inst/apps/SACH_hqidlsachap01/logs/appl/rgf/TXK/txkChkFormsDeployment_Thu_Jul_21_05_45_32_2016/txkChkFormsDeployment_Thu_Jul_21_05_45_32_2016.log

.end err out.

****************************************************


Executing service control script:
/u001/SACH/inst/apps/SACH_hqidlsachap01/admin/scripts/adoafmctl.sh start
script returned:
****************************************************

You are running adoafmctl.sh version 120.8

Starting OPMN managed OAFM OC4J instance  ...

adoafmctl.sh: exiting with status 150

adoafmctl.sh: check the logfile /u001/SACH/inst/apps/SACH_hqidlsachap01/logs/appl/admin/log/adoafmctl.txt for more information ...


.end std out.

.end err out.

Log file has below details 

  Root Service                                                                                                                 Enabled
  Root Service               Oracle Process Manager                                adopmnctl.sh     Started
  Web Entry Point Services                                                                                             Enabled
  Web Entry Point Services   Oracle HTTP Server                             adapcctl.sh          Failed
  Web Entry Point Services   OracleTNSListener                               adalnctl.sh           Started
  Web Application Services                                                                 Enabled
  Web Application Services   OACORE OC4J Instance                     adoacorectl.sh     Failed
  Web Application Services   FORMS OC4J Instance                        adformsctl.sh       Started
  Web Application Services   OAFM OC4J Instance                          adoafmctl.sh        Failed
  Batch Processing Services                                                                                             Disabled

• Deleted  “httpd.pid”  at   /u001/SACH/inst/apps/SACH_hqidlsachap01/pids/10.1.3/Apache  and   “.opmndat”  at  /oracle/apps/VIS/inst/apps/<CONTEXT>/ora/10.1.3/opmn/logs/states

• cleaned persistant directory for oacore , oafm and forms and restarted application and database still unable to start ebiz

• NFS share mounted on hqidlsachap01 is also being used by other servers,

• Observe rpciod deadlock on other server. periodically the rpciod is going into D state i.e uninterruptible sleep state.

• Load average and iowait% is also very high on these servers. 

• Finally rebooted all servers having common NFS and restarted application service to resolve the issue.