Monitoring of connection leaks in a web application is frequently overlooked. Relying on Throwables and finally blocks is not a sufficient enough remedy. Exceptions could still result in abandoned or unreturned connections. There is no good way to fix this on a case by case basis. A reasonable solution is to put a monitor in place that can clean up resources globally at some logical end points such as the end of a thread.
Assuming that connections are dedicated for a single "thread" of execution we can register all used connections on that thread and at the termination of the thread we can look at the connections that haven't been returned. We can log that abnormality and return those connections to the pool.
To that end we can use the following simple algorithm:
Although there are perhaps a number of ways for implementing this algorithm, I ended up choosing server side events. Under this approach the Connection Pool main line code will never need to be aware that connection monitoring is happening when it gives out a connection or gets a connection back.
Same thing with the http thread. The thread doesn't know that monitoring is taking place. Such functionality is delegated down to "specialized" handlers responding to these events. Orchestration of these handlers will get the job done. In a sense the monitoring "aspect" is managed by "out-of-band" code or "non-linear" code. What follows is annotated source code of how this scheme is implemented.
The following inteface outlines the events that happens in the life of a connection. These methods indicate which ones need to be specialized using inherited handlers.
public interface IConnectionEvents
{
public boolean onCreateConnection(Connection con) throws DBException;
public boolean onPreCloseConnection(Connection con) throws DBException;
public boolean onGetConnection(Connection con) throws DBException;
public boolean onPutConnection(Connection con) throws DBException;
}
As a pre-requisite the underlying connection pool need to honor these events and invoke them at appropriate times. This will give us a place holder to write extensible code for handling each of the specific events.
In a similar manner let us take a look at the Http events, the only other events interface that is needed, to see how we can intercept the end of an http thread.
public interface IHttpEvents
{
public boolean applicationStart() throws AspireServletException;
public boolean applicationStop() throws AspireServletException;
public boolean sessionStart(HttpSession session,
HttpServletRequest request,
HttpServletResponse response)
throws AspireServletException;
public boolean sessionStop() throws AspireServletException;
public boolean beginRequest(HttpServletRequest request,
HttpServletResponse response)
throws AspireServletException;
public boolean endRequest(HttpServletRequest request,
HttpServletResponse response)
throws AspireServletException;
public boolean userLogin(String username,
HttpSession session,
HttpServletRequest request,
HttpServletResponse response)
throws AspireServletException;
public boolean userChange(String oldUser,
String newUser, HttpSession session,
HttpServletRequest request, HttpServletResponse response)
throws AspireServletException;
}
The http event structure allows a number of methods to deal with extensible code with out interfering with the main line of code execution.
Connection monitoring can be implemented by overriding the appropriate methods of these two interfaces and interacting with a thread local variable. The high level conceptual design is illustrated in the following diagram.
The two event handlers request a static class SWIResourceCleanup to add, remove, and cleanup connections at appropriate times. SWIResourceCleanup holds a reference to a regular class ResourceCleanupRegistry in a ThreadLocal. This registry is set and reset once per each thread of execution. ResourceCleanupRegistry is responsible for asking the ConnectionPool at the end to close any unclosed or abandonded connections. SWIResourceCleanup ensures that the ThreadLocal is initialized and cleaned up for each thread of execution. Please note that only essential classes, not all classes, are represented in this diagram.
In "onGetConnection" an SQL connection is wrapped in a closable resource and passed to the SWIResourceCleanup. SWIResourceCleanup will figure out a way to hold this connection on a thread local for future reference. In "onPutConnection" we want to first locate the related Closable Resource and then remove it from the Resource Cleanup Registry. Let us examine the source code for onGetConnection and on putConnection.
//This is an implementation of the IConnectionEvents.
//Responsible for registering the connection in OnGetConnection.
//Responsible for unregisterting the connection in onPutConnection.
//Translates an SQL connection to a ClosableResource for
//registration and removal.
public class ConnectionCleanupHandler extends IConnectionEvents
{
//This map variable is needed to map between an SQL connection
//and a ClosableResource for removal purposes
private Hashtable ht = new Hashtable();
//.....some base class default implementations
//.....or you can inherit these from a default implementation
//Register the connection with a cleanup registry.
//connection is first converted to an "IClosableReource"
//and then registered.
//SWIResourceCleanup is a static class to facilitate this.
public boolean onGetConnection(Connection con) throws DBException
{
try
{
//Translate an SQL connection to an IClosableResource
ConnectionResource cr = new ConnectionResource(con);
//Remember the corresponding closable resource
//for removal purposes when this sql connection is
//returned.
ht.put(con, cr);
SWIResourceCleanup.addResource(cr);
return true;
}
catch(CommonException x)
{
throw new DBException("...some message...",x);
}
}
//Remove the connection from the registry.
//Locate the corresponding Closable Resource.
//Use that Closable Resource reference to remove.
public boolean onPutConnection(Connection con) throws DBException
{
try
{
//Locate the corresponding closable resourcee
ConnectionResource cr = (ConnectionResource)ht.get(con);
if (cr == null)
{
AppObjects.trace(this, "Connection close wrapper not found!!");
return true;
}
//connection wrapper is there
SWIResourceCleanup.removeResource(cr);
return true;
}
catch(CommonException x)
{
throw new DBException("...some message..",x);
}
finally
{
//Either way remove that connection reference
AppObjects.trace(this,"Removing the connection from local registry");
this.ht.remove(con);
}
}
}//eof-class
The SWIResourceCleanup is a static class that can work with any resource that is closable. It only deals with IClosableResource interface. It is not aware of a "connection". The class "ConnectionResource" wraps a "connection" and passes it to the SWIResourceCleanup. The prefix "SWI" stands for a "Static Wrapper to an Interface". It is a static class that makes it convenient for the clients to call easily with out worrying about obtaining a reference to an interface. When you are removing the connection from the registry it is important to remember the wrapper class that wrapped that connection. To accomodate this a hashtable is used as an intermediate a look up.
Let us take a closer look at the ConnectionResource class which is essentially a "ClosableResoruce" with a "close" method on it.
public class ConnectionResource implements IClosableResource
{
private Connection m_con = null;
public ConnectionResource(Connection con)
{
m_con = con;
}
//inherited method from the interface
public void close() throws DataException
{
...
DBUtils.putConnection(m_con);
...
}
}//eof-class
Now that we have sucessfully registered connections we need to tally them and close them if needed when the http thread comes to a close. To do this we need to implement the "endRequest" method from the IHttpEvents interface.
public class ResourceCleanupHandler extends DefaultHttpEvents
{
//Is invoked in the "finally" block of the servlet "run" method
public boolean endRequest(HttpServletRequest request,
HttpServletResponse response)
throws AspireServletException
{
......
SWIResourceCleanup.cleanup();
return true;
....
}
}//eof-class
This class is responsible for implementing the outlined protocol. This class defines what it means to add a resource or remove a resource or cleanup the resources at the end of an http thread.
public class ResourceCleanupRegistry implements IResourceCleanupRegistry
{
//A list to hold closable resources.
//List
private List m_resourceList = new ArrayList();
public ResourceCleanupRegistry(){}
//Add a resource to the list.
public void addResource(IClosableResource icr)
throws CommonException
{
m_resourceList.add(icr);
}
//Remove a resource from the list.
public void removeResource(IClosableResource icr)
throws CommonException
{
m_resourceList.remove(icr);
}
//Log the number of resources that are not cleaned up.
//For each resource call a close on them.
public void cleanup() throws CommonException
{
//See how many resources are available.
int numOfResources = this.m_resourceList.size();
AppObjects.trace(this, "Number of unclosed resources:" + numOfResources);
if (numOfResources == 0)
{
return;
}
//There are resources to be cleaned up.
//You may have to copy the list so that a close won't alter this list
//through self generated event.
List clonedList = new ArrayList();
clonedList.addAll(m_resourceList);
Iterator itr = clonedList.iterator();
while(itr.hasNext())
{
IClosableResource icr =
(IClosableResource)itr.next();
icr.close();
}
//By now the m_resourceList should be empty.
numOfResources = m_resourceList.size();
if (numOfResources > 0)
{
AppObjects.warn(this,"...some message...");
}
}
}//eof-class
However this class by itself will be effete unless someone places it on a thread local and orchestrates that ThreadLocal behavior. The earlier mentioned static wrapper does that. Each thread holds reference to one of these "Registry" objects. Each Registry object in turn holds a number of connections as they get to be used on this thread. The need now is to write the code to place this registry on a thread using thread loal concept.
SWIResourceCleanup is a static class that holds a thread local variable and stores the registry in that variable. This thread local is initialized the first time a connection is obtained on this thread. The registry is removed when the thread ends by setting the thread local to null during the clean up.
public class SWIResourceCleanup
{
//Set up a static variable for the thread local.
//Do not set its contents in a static initialization!!
//That will defeat the purpose.
public static ThreadLocal m_ResourceRegistryTL = new ThreadLocal();
//helper method to get the registry object from thread local
private static IResourceCleanupRegistry getRegistry()
{
return (IResourceCleanupRegistry)m_ResourceRegistryTL.get();
}
//Make sure this variable is set on each thread
//at the begining.
private static void setRegistryOnThisThread()
{
IResourceCleanupRegistry ircr = getRegistry();
if (ircr == null)
{
AppObjects.trace("com.ai.resourcecleanup.SWIResourceCleanup"
,"Setting up registry for this thread");
m_ResourceRegistryTL.set(new ResourceCleanupRegistry());
}
}
//At the end remove the registry by setting its contents to null
private static void resetRegistry()
{
AppObjects.trace("com.ai.resourcecleanup.SWIResourceCleanup"
,"Resetting Registry to null for this thread");
m_ResourceRegistryTL.set(null);
}
//Delegate the method to the underlying registry
public static void addResource(IClosableResource icr)
throws CommonException
{
setRegistryOnThisThread();
getRegistry().addResource(icr);
}
//Delegate the method to the underlying registry
public static void removeResource(IClosableResource icr)
throws CommonException
{
IResourceCleanupRegistry ircr = getRegistry();
if (ircr == null)
{
AppObjects.warn("com.ai.resourcecleanup.SWIResourceCleanup"
,"No registry to remove the resource from.");
return;
}
getRegistry().removeResource(icr);
}
//Responsible for cleaning up the connections.
//Delegate the method to the underlying registry.
//Important: make sure the ThreadLocal is cleaned up.
//ThreadLocal has to be set to null in the end.
public static void cleanup()
throws CommonException
{
IResourceCleanupRegistry ircr = getRegistry();
if (ircr == null)
{
AppObjects.warn("com.ai.resourcecleanup.SWIResourceCleanup"
,"No registry to cleanup.");
return;
}
getRegistry().cleanup();
//After cleanup remember to set the registry to null on this thread
resetRegistry();
}
}//eof-class
For a broader understanding of server side events you can read this java.net article Understanding Server side Event Distributors . However for the discussion here, the magic happens through a configuration file that looks something like the following:
# SQL Connection events
request.aspire.db.connectionevents.classname=\
com.ai.db.events.ConnectionEventDistributor
request.aspire.db.connectionevents.eventHandlerList=\
ConnectionCleanupHandler
request.ConnectionCleanupHandler.classname=\
com.ai.resourcecleanup.ConnectionCleanupHandler
The "endRequest" handler also gets called in a similar fashion
# HttpEvents
request.IHttpEvents.classname=\
com.ai.servlets.HttpEventDistributor
request.IHttpEvents.eventHandlerList=\
defaultHandler,ResourceCleanupHandler
request.defaultHandler.classname=\
com.ai.servlets.DefaultHttpEvents1
request.ResourceCleanupHandler.classname=\
com.ai.servlets.ResourceCleanupHandler
Here is a picture of how "HttpEvents" work
One can use the following handy rules to quickly check the connection handling maturity of a web application.
I can imagine one using aspect oriented programming instead of connection events to accomplish a similar goal. It would be a good exercise.
It is also possible to have the connection pool use some kind of time limits on abandoned connections to bring them back to the available pool or close them.
As part of the event processing it is also possible to rewrite the distributor pattern using dynamic proxies as opposed to implementing each distributor by hand.
1. Understanding Server side Event Distributors