Why do they get Hung?
The main reason the threads hang is because it can not finish the request in a timely manner. If your data needs continuously updating, some of queries loose efficiency because their execution paths over time need to be re-analyzed. Over the last couple years we have seen this in several of our queries.

Since we have a set number of available DRP threads (40 in our case), when they are all hanging, the application server is no longer able to serve requests.

This problem is an Enterprise problem. This means all ATG applications run into this problem, if the queries don’t return.

The solution to the problem.

Overview
The solution is to first monitor and identify DRP hung threads, send email regarding their hung status, and then kill them before they can do damage. This solution requires a monitor running inside the application server that will periodically take an inventory of the running DRP threads. When a thread age reaches a predetermined threshold, it will be terminated, and information about the terminated thread will be sent to a predetermined email for evaluation.

The following are the control inputs for the DRPThread Monitor stored in a property file. The property file is loaded upon startup.

This tells the monitor to auto-start.
threadMonitorAutoStart = true

This is how often it checks the DRP threads
threadMonitorWaitInterval = 5000

This is percent of DRP threads in use before checking threshold
threadMonitorThreshold= 80

This is the Age limit of threads. When the age limit is surpassed
thresholdAgeLimit = 300000

This is the max age of any thread. A thread over this age is killed
maxThreadAge= 1800000

List of email address sent notifications when a DRP thread is killed
emailNotificationList = hwilliamson@greattastingjava.com

Pseudo Code

1. Application server starts up
2. Starts DRP Monitor

a. If threadMonitorAutoStart = true, then monitor continues to start up, else it stops.
b. Loop until forced stop

i. Check DRP server
1. percent active thread > threadMonitorThreshold && age of thread > thresholdAgeLimit) || age of thread > maxThreadAge then
a. Email info about hung thread
b. Kill thread
ii. Sleep threadMonitorWaitInterval

Code Sample

while (!forcedStop) {

try {

RequestServerHandler handlers[] = drpServer.getRequestHandlers();

int activeCount = drpServer.getActiveHandlerCount();

double percentActive = (activeCount/handlers.length) *100.0;

for (int i = 0; i < handlers.length; i++) {

if (handlers[i] instanceof DrpServerConnection) {

DrpServerConnection connection = (DrpServerConnection) handlers[i];
if (connection.getHandlingRequest()) {

if ((percentActive> threadMonitorThreshold && connection.getCurrentRequestTime() > thresholdAgeLimit) ||connection.getCurrentRequestTime() > maxThreadAge)
{
logger.debug(”killing:” + connection);

String path = connection.getCurrentRequestPathInfo();
long time = connection.getCurrentRequestTime();

connection.killHandler();

Mail mail = new Mail();
mail.postMail(emailNotificationList,”DRP Thread Killed”, “Path:” +path +”\n”
+”Time:” +time+”\n”,”hwilliamson@greattastingjava.com”);
logger.debug(”emailing:” + emailNotificationList);

}
}
}
}
sleepDRPThreadMonitor();
} catch (Exception ex)
{
ex.printStackTrace();
}
}

Technorati Tags: Spring, Java, framework, helloword, code example

a. Alias: headleyalias (made up name used later)
b. Username: headleyuser (Database user name)
c. Password: headleyuser (Database password)

9. Now under your “server” tab, deploy/publish your EAR and start your the Websphere 6.0 development environment
10. Right click the server and select “Run the administration console”. This will start admin console in a new window.
11. Click Login (any login will work)
12. Click “Resources”
13. Click “JDBC Providers”
14. Scroll to bottom of display and click “new”, and fill out the following fields

a. Name: Oracle JDBC Driver
b. Description: Oracle JDBC Driver
c. Class path: C:/Oracle/Ora92/jdbc/lib/classes12.zip (on my machine)
d. Implementation class name: oracle.jdbc.pool.OracleConnectionPoolDataSource
e. Apply and OK
f. Don’t forget to save.

15. Now from the JDBC provider window click “Oracle JDBC Driver” which you just created.
16. Now on the right side of the window under “Additional properties”, click “data sources”.
17. Click “new”, and fill out the following fields

a. Name: Oracle JDBC Driver DataSource
b. JNDI name: jdbc/myapp ( This is the JNDI name you need to use to look up this data source)
c. Data store class name: Select a data store helper -> Oracle 9i and prior data store helper
d. Component-managed authentication alias: select “headleyalias” (your alias)
e. Component-managed authentication: select “headleyalias” (your alias)
f. URL: jdbc:oracle:thin:@servername:1521:databasename
g. Test your connection
h. Apply and OK
i. Don’t forget to save.

18. Now under your “serv

er” tab, deploy/publish your EAR and restart your the Websphere 6.0 development environment.

This is what I found:

Bean Managed Transaction in Message Beans:
• Single threaded if more than one data source was used.

Container Managed Transaction in Message Beans:
• Multi threaded even if more than one data source was used, but slower due to XA Transactions.

The solution I found:
Use Container Management Transactions, but use a stateless session bean with bean managed transactions to access your second data source.

Message Beans, Java, MQ, Transactions

Technorati Tags: Java, ATG, Performance, Optimizeit , Dynamo

Technorati Tags: Java, JNDI, Websphere, Websphere 6.0, MQ, Websphere MQ, Websphere MQ 5.3

As you go through this tutorial there are things to keep in hand

• Queue Manager Name : QM_headley (default on my machine)
• Queue Manager Port :1414 (default)
• Queue name: headleyqueue (created this under QM_headley in MQ explorer )
• File System url : file:/C:/JNDI-Directory/mq
• File System JNDI Connection Factory name : headleyCF
• File System JNDI Queue name : headleyq
• Websphere Provider Connection Factory name: jms/headleyCF
• Websphere Provider Queue name: jms/headleyq
• Listener port name: headleyListener
• Test Java code: QueueTest.java QueuePut.java

In MQ 5.3 explorer

• Create Queue Manager or Use default Queue Manager
• Create a queue in MQ series i.e. “headleyqueue“

Use there MQ’s JMSAdmin tool to set up bindings for the Queue

• It can be found under -> C:\Program Files\IBM\WebSphere MQ\Java\bin
• Edit JMSAdmin.config to be a file system provider set the follwoing

• INITIAL_CONTEXT_FACTORY=com.sun.jndi.fscontext.RefFSContextFactory
• PROVIDER_URL=file:/C:/JNDI-Directory
• SECURITY_AUTHENTICATION=none

Test queue with this sample application

QueueTest -url file:/C:/JNDI-Directory/mq (you need to have all the jar file in C:\Program Files\IBM\WebSphere MQ\Java\lib in your classpath to run)
In Rational Software Development Platform Test Environment 6.0

• Create Test Project
• Create Test Package
• Create Test MessageBean
• Create Test Server
• Add Bean to Websphere Test Server
• Start Websphere Test Server
• Run Admin console and select server1 by right clicking server

Login into Admin Console with any ID

• Configure Websphere MQ JMS provider

• Configure “Websphere MQ queue connection factories”under Resources/JMS providers/Websphere MQ/ in the Additional properties section (second one from the top)

Click new

Fill out the following fields

Name: headleyCF (made up name)

JNDI Name: jms/headleyCF (made up JNDI name)

Queue Manager: QM_headley (default)

Host: localhost

Port :1414

Don’t forget to SAVE after pressing Apply and OK

• Configure “Websphere MQ queue destinations factories”under Resources/JMS providers/Websphere MQ/ in the Additional properties section (third one from the top)
• Click new

Fill out the following fields

Name: headleyCF (made up name)

JNDI Name: jms/headleyq (made up JNDI name)

Base Queue name: headleyqueue (from MQ series)

Base Queue Manager name: QM_headley (default on my machine)

Queue Manager Host: localhost

Queue Manager Port :1414

Don’t forget to SAVE after pressing Apply and OK

• Configure Listener Ports with JNDI names from Connection Factory and JMS Destinations created in previous two steps.

First select server1 under Servers/ApplicationServers.

Then under communications select messaging/ message listening service

Then click Listener Ports

Then Click new

Fill out the following fields

Name: headleyListener (made up name)

Connection Factory JNDI Name: jms/headleyCF

Destination JNDI name: jms/headleyq

Don’t forget to SAVE after pressing Apply and OK

Do not try to start Listener, it will start when you restart the server

• Configure Message Bean with Listener Ports

In J2ee view double click bean and fill in the Listener port Name: headleyListener

• Restart Server

Test with MessageBean Test

QueuePut -url file:/C:/JNDI-Directory/mq (you need to have all the jar files in C:\Program Files\IBM\WebSphere MQ\Java\lib in your classpath to run)

Technorati Tags: Websphere MQ, Webphere MQ 5.3, MQ, Rational Development Environment 6.0, Rational, Websphere, Webphere 6.0