Why do they get Hung?
The main reason the threads hang is because it can not finish the request in a timely manner. If your data needs continuously updating, some of queries loose efficiency because their execution paths over time need to be re-analyzed. Over the last couple years we have seen this in several of our queries.

Since we have a set number of available DRP threads (40 in our case), when they are all hanging, the application server is no longer able to serve requests.

This problem is an Enterprise problem. This means all ATG applications run into this problem, if the queries don’t return.

The solution to the problem.

Overview
The solution is to first monitor and identify DRP hung threads, send email regarding their hung status, and then kill them before they can do damage. This solution requires a monitor running inside the application server that will periodically take an inventory of the running DRP threads. When a thread age reaches a predetermined threshold, it will be terminated, and information about the terminated thread will be sent to a predetermined email for evaluation.

The following are the control inputs for the DRPThread Monitor stored in a property file. The property file is loaded upon startup.

This tells the monitor to auto-start.
threadMonitorAutoStart = true

This is how often it checks the DRP threads
threadMonitorWaitInterval = 5000

This is percent of DRP threads in use before checking threshold
threadMonitorThreshold= 80

This is the Age limit of threads. When the age limit is surpassed
thresholdAgeLimit = 300000

This is the max age of any thread. A thread over this age is killed
maxThreadAge= 1800000

List of email address sent notifications when a DRP thread is killed
emailNotificationList = hwilliamson@greattastingjava.com

Pseudo Code

1. Application server starts up
2. Starts DRP Monitor

a. If threadMonitorAutoStart = true, then monitor continues to start up, else it stops.
b. Loop until forced stop

i. Check DRP server
1. percent active thread > threadMonitorThreshold && age of thread > thresholdAgeLimit) || age of thread > maxThreadAge then
a. Email info about hung thread
b. Kill thread
ii. Sleep threadMonitorWaitInterval

Code Sample

while (!forcedStop) {

try {

RequestServerHandler handlers[] = drpServer.getRequestHandlers();

int activeCount = drpServer.getActiveHandlerCount();

double percentActive = (activeCount/handlers.length) *100.0;

for (int i = 0; i < handlers.length; i++) {

if (handlers[i] instanceof DrpServerConnection) {

DrpServerConnection connection = (DrpServerConnection) handlers[i];
if (connection.getHandlingRequest()) {

if ((percentActive> threadMonitorThreshold && connection.getCurrentRequestTime() > thresholdAgeLimit) ||connection.getCurrentRequestTime() > maxThreadAge)
{
logger.debug(”killing:” + connection);

String path = connection.getCurrentRequestPathInfo();
long time = connection.getCurrentRequestTime();

connection.killHandler();

Mail mail = new Mail();
mail.postMail(emailNotificationList,”DRP Thread Killed”, “Path:” +path +”\n”
+”Time:” +time+”\n”,”hwilliamson@greattastingjava.com”);
logger.debug(”emailing:” + emailNotificationList);

}
}
}
}
sleepDRPThreadMonitor();
} catch (Exception ex)
{
ex.printStackTrace();
}
}

Sokkit: 4.0
JDK: 1.5.0_05
Tomcat: 5.0.28

1. Download and unzip the sokkit_tomcat kit
2. Shutdown apache
3. Download and unzip tomcat.

http://archive.apache.org/dist/jakarta/tomcat-5/v5.0.28/bin/

4. Download and install JDK 1.5.

http://java.sun.com

5. Copy “jakarta-tomcat-5.0.28″ to sokkit folder
6. Added
   

   set JAVA_HOME=C:\\Program Files\\Java\\jdk1.5.0_05

   (Where ever you have your JDK installed)
   to top of bin\catalina.bat (after comments)

7. Copy tomcat.conf to C:\sokkit\apache2\conf\addonconf
8. Copy mod_jk.so in C:\sokkit\apache2\modules folder
9. Start tomcat

a. cd C:\\sokkit\\jakarta-tomcat-5.0.28\\bin b. startup

10. Start apache
11. Test page http://localhost/servlets-examples/