A Clustered ClassLoader

If you're building a distributed system, or contemplating building a distributed system, you might have run into this one before:

1. You write and compile your classes in Eclipse
   
2. You try out your classes on your laptop -- they work (woohoo!)
   
3. Its a distributed system so you need to make sure your classes work in a true distributed environment
   
4. You publish your classes to the distributed systems
   
5. You try out your classes -- and they don't work (boo!)
   
6. You fix the problem.
   
7. You publish the classes again.
   
8. Rinse, repeat.
   

After doing this a few dozen times, you find that publishing your classes to distributed systems is a total PITA that you would rather avoid altogether.

Or, you might have an application, like Master/Worker in which you deploy some part of the application at deploy time, but you deploy other parts of it during run-time. In the Master/Worker case, you deploy the Master and the Worker, but the Work comes and goes, and you'd like to be able to deploy new Work easily and trivially. In the Master/Worker case, since Masters are usually in control, and there is a farm of Workers, you'd like to deploy some new work to the Master, and let it send the Work to the Workers. Knowing about the Work up front on the Workers is a non-starter.

Some solutions to this problem?

• Java has dynamic code loading capabilities already. Deploy your class files to a shared filesystem like NFS, and deploy your code to a shared directory.
  
• Java also supports loading code from URLs (thanks to it's Applet heritage) so deploy your code to an HTTP server and you're set
  
• Factor your application such that new classes aren't needed - just make the new definitions "data" driven
  
• Embed a scripting engine, so you can pass Strings and interpret them as code - BeanShell, Jython, JRuby, Javascript, and Groovy all come to mind here...
  

Those are all fine solutions, but it never hurts to have more tools in your toolbox does it? Especially if you're already using Terracotta, wouldn't it be nice if there was some way to just leverage Terracotta's core clustering capabilities to build a clustered classloader?

I've done just that. Here's how it works:

• Your application tries to instantiate a class, which means it asks the currently in scope ClassLoader to instantiate the class (by name)
  
• By launching the application under the clustered classloader, it is in scope.
  
• The clustered class loader has a Map<String, byte[]> that correlates classnames to bytes
  
• The clustered class loader looks in this Map, if the classname is found, it uses the byte[] to create the requested class using defineClass()
  
• If the class wasn't found in the Map, then it looks in the filesystem to find the class
  
• If the class bytes are found on the filesystem, then it reads them into a byte[], and stashes them in the clustered Map<String, byte[]>
  
• If the bytes aren't found, it just delegates to the parent classloader
  

I've omitted some of the finer details. The Map used is actually a Map<String, ClassMetaData> where ClassMetaData is a class that holds a long modified and byte[] bytes.

Let's have a look at the important parts of the ClusterClassLoader:

public class ClusterClassLoader extends ClassLoader
{
    private static final String NAME = "ClusterClassLoader";
    
    private static Map<String, Class> classes = new HashMap<String, Class>();
    private static Map<String, ClassMetaData> bytes = new HashMap<String, ClassMetaData>();
    private static transient boolean loaded;

    ...

ClusterClassLoader is defined to extend ClassLoader. It has a NAME field, which will be used to give a name to this classloader. This is a requirement for a classloader used by Terracotta. Normally Terracotta does this for you, but we are defining a new classloader, so we have to follow the naming rules for Terracotta (naming gives ClassLoaders across the cluster a unique identity).

A classes field is defined, which caches the result of the defineClass operation in the local JVM only. A bytes field is defined. This field is marked as a root, so that it can be shared with every other instance of ClusterClassLoader in the cluster.

The constructor detects if Terracotta is loaded using some reflection, and if so registers the classloader and sets a flag to enable cluster classloading features:

    public ClusterClassLoader()
    {
        super(ClusterClassLoader.class.getClassLoader());
        try {
            Class namedClassLoader = findClass("com.tc.object.loaders.NamedClassLoader");
            Class helper = findClass("com.tc.object.bytecode.hook.impl.ClassProcessorHelper");
            Method m = helper.getMethod("registerGlobalLoader", new Class[] { namedClassLoader }); 
            m.invoke(null, new Object[] { this });
            loaded = true;
        } catch (Exception e) {
            // tc is not present, so don't do anything fancy
            loaded = false;
        }
    }

Next, the definition of loadClass is overridden:

    @Override
    public Class<?> loadClass(String name) throws ClassNotFoundException
    {
        return findClass(name);
    }

and so is findClass:

     @Override
    protected Class<?> findClass(String name) throws ClassNotFoundException
    {        
        if (!loaded) {
            return getParent().loadClass(name);            
        }
        
        Class result = null;
        synchronized (classes) {
            result = classes.get(name);
            if (result != null) { return result; }

            result = loadClassBytes(name);
            if (result == null) { return getParent().loadClass(name); }
            classes.put(name, result);
        }
        
        return result;
    }

This is the bulk of the algorithm. The loaded flag is set when the class loader is instantiated. It used a bit of reflection to determine if Terracotta was even present in the JVM. If not, it is set to false, and the ClusterClassLoader just delegates to the parent class loader.

If Terracotta is present, then it checks to see if the class has already been defined. If so, it is returned directly from the classes cache. If it has not, then it gets the bytes from the loadClassBytes method. If that cannot find the bytes, then it asks the parent class loader to load the class.

The bulk of the implementation is done in the loadClassBytes method:

    private Class loadClassBytes(String name) throws ClassNotFoundException
    {
        ClassMetaData metaData;
        
        synchronized (bytes) {        
            try {
                File f = null;
                metaData = bytes.get(name);
                URL resource = ClassLoader.getSystemResource(name.replace('.',File.separatorChar)+".class");
                // if resource is non null, then the class is on the local fs (in the cp)
                if (resource != null) {
                    f = new File(resource.getFile());        
                }
                
               if (metaData != null) {
                   // if it's cached, but not on the fs, return it.
                   // if it's cached, but on the fs, check to see if it's 
                   // up to date
                   if (f == null || metaData.modified >= f.lastModified()) { 
                       return defineClass(name, metaData.bytes, 0, metaData.bytes.length, null);
                   }
                }
                
                // load from the fs
                byte[] classBytes = loadClassData(f);
                Class result =  defineClass(name, classBytes, 0, classBytes.length, null);

                try {
                    result.getDeclaredField("$__tc_MANAGED");
                    // it's managed so cache it
                    bytes.put(name, new ClassMetaData(f.lastModified(), classBytes));
                } catch (NoSuchFieldException e) {
                    // not managed don't cache it
                }
                return result;
            } catch (IOException e){
                return null;
            } 
        }
    }    

This method looks for the cached bytes, and for a file that corresponds to the class. If both are found, then it compares the modified date of the two. If the modified date of the bytes are greater than or equal to the file, then it returns the bytes in the cache. Otherwise it loads the bytes from the file. Once the bytes are loaded, defineClass is called to turn the bytes into a class file.

At this point, the ClusterClassLoader can check to see if the class is instrumented by Terracotta. Every instance of a class that is shared by Terracotta must be instrumented, so it's not necessary to cache class bytes for classes that are not instrumented by Terracotta. If the class is instrumented by Terracotta, then the ClusteredClassLoader stashes the bytes into the class bytes cache.

Click here if you would like to see the source code to ClusterClassLoader in its entirety

UPDATE: This project has been included in the tim-tclib project, and is a runnable sample. More details can be found in the sample readme.html

I've put the whole thing together as a simple runnable example. You just have to check out the source for the project, and run a few simple Maven commands. You can get the demo from here:

$ svn checkout http://svn.terracotta.org/svn/forge/projects/labs/tim-clusterclassloader clusterclassloader
$ cd clusterclassloader

The demo defines a main project, and two sub projects. The first sub project, sample, is responsible for putting classes into a queue. The second sub project, sample2, reads from the queue. To test the effectiveness of the cluster class loader, the second sample of course does not have the classes from the first sub project.

To run the demo:

1. Build the project:

   
   $ mvn install

2. Cd to the sample directory, compile and start a tc server:

   
   $ cd sample
   $ mvn package
   $ mvn tc:start

3. Start the sample process:

   
   $ mvn tc:run
   

4. In another terminal, cd to the sample2 directory:

   
   $ cd sample2

5. Compile, and run the example:

   
   $ mvn package
   $ mvn tc:run

   
   

If you did everything correctly, you should see:

[INFO] [node] Waiting for work...
[INFO] [node] This is Callable2 calling!

In the second terminal (sample2). The message printed ("This is Callable2 calling!") is printed by a class that is only present in the classpath of the first instance (sample).