Saturday, March 22, 2008

A Clustered ClassLoader

If you're building a distributed system, or contemplating building a distributed system, you might have run into this one before:


  1. You write and compile your classes in Eclipse
  2. You try out your classes on your laptop -- they work (woohoo!)
  3. Its a distributed system so you need to make sure your classes work in a true distributed environment
  4. You publish your classes to the distributed systems
  5. You try out your classes -- and they don't work (boo!)
  6. You fix the problem.
  7. You publish the classes again.
  8. Rinse, repeat.


After doing this a few dozen times, you find that publishing your classes to distributed systems is a total PITA that you would rather avoid altogether.

Or, you might have an application, like Master/Worker in which you deploy some part of the application at deploy time, but you deploy other parts of it during run-time. In the Master/Worker case, you deploy the Master and the Worker, but the Work comes and goes, and you'd like to be able to deploy new Work easily and trivially. In the Master/Worker case, since Masters are usually in control, and there is a farm of Workers, you'd like to deploy some new work to the Master, and let it send the Work to the Workers. Knowing about the Work up front on the Workers is a non-starter.

Some solutions to this problem?

  • Java has dynamic code loading capabilities already. Deploy your class files to a shared filesystem like NFS, and deploy your code to a shared directory.
  • Java also supports loading code from URLs (thanks to it's Applet heritage) so deploy your code to an HTTP server and you're set
  • Factor your application such that new classes aren't needed - just make the new definitions "data" driven
  • Embed a scripting engine, so you can pass Strings and interpret them as code - BeanShell, Jython, JRuby, Javascript, and Groovy all come to mind here...


Those are all fine solutions, but it never hurts to have more tools in your toolbox does it? Especially if you're already using Terracotta, wouldn't it be nice if there was some way to just leverage Terracotta's core clustering capabilities to build a clustered classloader?

I've done just that. Here's how it works:

  • Your application tries to instantiate a class, which means it asks the currently in scope ClassLoader to instantiate the class (by name)
  • By launching the application under the clustered classloader, it is in scope.
  • The clustered class loader has a Map<String, byte[]> that correlates classnames to bytes
  • The clustered class loader looks in this Map, if the classname is found, it uses the byte[] to create the requested class using defineClass()
  • If the class wasn't found in the Map, then it looks in the filesystem to find the class
  • If the class bytes are found on the filesystem, then it reads them into a byte[], and stashes them in the clustered Map<String, byte[]>
  • If the bytes aren't found, it just delegates to the parent classloader


I've omitted some of the finer details. The Map used is actually a Map<String, ClassMetaData> where ClassMetaData is a class that holds a long modified and byte[] bytes.

Let's have a look at the important parts of the ClusterClassLoader:

public class ClusterClassLoader extends ClassLoader
{
private static final String NAME = "ClusterClassLoader";

private static Map<String, Class> classes = new HashMap<String, Class>();
private static Map<String, ClassMetaData> bytes = new HashMap<String, ClassMetaData>();
private static transient boolean loaded;

...
ClusterClassLoader is defined to extend ClassLoader. It has a NAME field, which will be used to give a name to this classloader. This is a requirement for a classloader used by Terracotta. Normally Terracotta does this for you, but we are defining a new classloader, so we have to follow the naming rules for Terracotta (naming gives ClassLoaders across the cluster a unique identity).

A classes field is defined, which caches the result of the defineClass operation in the local JVM only. A bytes field is defined. This field is marked as a root, so that it can be shared with every other instance of ClusterClassLoader in the cluster.

The constructor detects if Terracotta is loaded using some reflection, and if so registers the classloader and sets a flag to enable cluster classloading features:
    public ClusterClassLoader()
{
super(ClusterClassLoader.class.getClassLoader());
try {
Class namedClassLoader = findClass("com.tc.object.loaders.NamedClassLoader");
Class helper = findClass("com.tc.object.bytecode.hook.impl.ClassProcessorHelper");
Method m = helper.getMethod("registerGlobalLoader", new Class[] { namedClassLoader });
m.invoke(null, new Object[] { this });
loaded = true;
} catch (Exception e) {
// tc is not present, so don't do anything fancy
loaded = false;
}
}


Next, the definition of loadClass is overridden:
    @Override
public Class<?> loadClass(String name) throws ClassNotFoundException
{
return findClass(name);
}
and so is findClass:
     @Override
protected Class<?> findClass(String name) throws ClassNotFoundException
{
if (!loaded) {
return getParent().loadClass(name);
}

Class result = null;
synchronized (classes) {
result = classes.get(name);
if (result != null) { return result; }

result = loadClassBytes(name);
if (result == null) { return getParent().loadClass(name); }
classes.put(name, result);
}

return result;
}
This is the bulk of the algorithm. The loaded flag is set when the class loader is instantiated. It used a bit of reflection to determine if Terracotta was even present in the JVM. If not, it is set to false, and the ClusterClassLoader just delegates to the parent class loader.

If Terracotta is present, then it checks to see if the class has already been defined. If so, it is returned directly from the classes cache. If it has not, then it gets the bytes from the loadClassBytes method. If that cannot find the bytes, then it asks the parent class loader to load the class.

The bulk of the implementation is done in the loadClassBytes method:

private Class loadClassBytes(String name) throws ClassNotFoundException
{
ClassMetaData metaData;

synchronized (bytes) {
try {
File f = null;
metaData = bytes.get(name);
URL resource = ClassLoader.getSystemResource(name.replace('.',File.separatorChar)+".class");
// if resource is non null, then the class is on the local fs (in the cp)
if (resource != null) {
f = new File(resource.getFile());
}

if (metaData != null) {
// if it's cached, but not on the fs, return it.
// if it's cached, but on the fs, check to see if it's
// up to date
if (f == null || metaData.modified >= f.lastModified()) {
return defineClass(name, metaData.bytes, 0, metaData.bytes.length, null);
}
}

// load from the fs
byte[] classBytes = loadClassData(f);
Class result = defineClass(name, classBytes, 0, classBytes.length, null);

try {
result.getDeclaredField("$__tc_MANAGED");
// it's managed so cache it
bytes.put(name, new ClassMetaData(f.lastModified(), classBytes));
} catch (NoSuchFieldException e) {
// not managed don't cache it
}
return result;
} catch (IOException e){
return null;
}
}
}
This method looks for the cached bytes, and for a file that corresponds to the class. If both are found, then it compares the modified date of the two. If the modified date of the bytes are greater than or equal to the file, then it returns the bytes in the cache. Otherwise it loads the bytes from the file. Once the bytes are loaded, defineClass is called to turn the bytes into a class file.

At this point, the ClusterClassLoader can check to see if the class is instrumented by Terracotta. Every instance of a class that is shared by Terracotta must be instrumented, so it's not necessary to cache class bytes for classes that are not instrumented by Terracotta. If the class is instrumented by Terracotta, then the ClusteredClassLoader stashes the bytes into the class bytes cache.

Click here if you would like to see the source code to ClusterClassLoader in its entirety

UPDATE: This project has been included in the tim-tclib project, and is a runnable sample. More details can be found in the sample readme.html

I've put the whole thing together as a simple runnable example. You just have to check out the source for the project, and run a few simple Maven commands. You can get the demo from here:

$ svn checkout http://svn.terracotta.org/svn/forge/projects/labs/tim-clusterclassloader clusterclassloader
$ cd clusterclassloader


The demo defines a main project, and two sub projects. The first sub project, sample, is responsible for putting classes into a queue. The second sub project, sample2, reads from the queue. To test the effectiveness of the cluster class loader, the second sample of course does not have the classes from the first sub project.

To run the demo:

  1. Build the project:

    $ mvn install
  2. Cd to the sample directory, compile and start a tc server:

    $ cd sample
    $ mvn package
    $ mvn tc:start
  3. Start the sample process:

    $ mvn tc:run
  4. In another terminal, cd to the sample2 directory:

    $ cd sample2
  5. Compile, and run the example:

    $ mvn package
    $ mvn tc:run



If you did everything correctly, you should see:

[INFO] [node] Waiting for work...
[INFO] [node] This is Callable2 calling!
In the second terminal (sample2). The message printed ("This is Callable2 calling!") is printed by a class that is only present in the classpath of the first instance (sample).

14 comments:

Sean Flanigan said...

Hi, I've been trying out the example, but so far it isn't working for me.

I found that I had to check out clusteredclassloader from a different URL (which I basically guessed):
$ svn co http://svn.terracotta.org/svn/forge/projects/labs/tim-clusterclassloader clusteredclassloader

I also needed tim-parent:
$ svn co http://svn.terracotta.org/svn/forge/projects/tim-parent/tags/release-1.0.5 tim-parent
$ cd tim-parent; mvn install
$ cd ../clusteredclassloader
$ mvn install

But when I go to run the samples, the first program exits after a few seconds. Then the second program fails:

[INFO] [node] Exception in thread "main" com.tc.exception.TCClassNotFoundException: java.lang.ClassNotFoundException: org.terracotta.sample.
work.MyCallable2

Looks like the Worker is getting the job from the queue, but not the class.

What's wrong? Have I grabbed the wrong version from SVN?

Thanks,

Sean.

Taylor said...

@Sean,

Thanks, and good guesses! I forgot to run in a clean environment, so I missed a setting in my main pom.xml.

I have updated the main pom, so the tim-parent checkout and install shouldn't be necessary any longer.

I wiped my .m2 repo and started over again, following my instructions to the letter, and it works for me.

So I am not sure why it doesn't work for you - if you start over -- wipe your .m2/repository/org/terracotta directory -- and it still doesn't work feel free to jump on our forums or email me : tgautier -at- terracotta.org.

David said...

When I tried this technique for naming my classloader, I got the following running on Linux (jdk 1.6.0_05): java.lang.ClassNotFoundException: com.tc.object.loaders.NamedClassLoader
at java.lang.ClassLoader.findClass(ClassLoader.java:358)
at com.my.platform.servicenode.core.MyClassLoader.< init >(MyClassLoader.java:35)
at com.my.platform.servicenode.core.ServiceNode.loadLibraries(ServiceNode.java:185)
at com.my.platform.servicenode.core.ServiceNode.__tc_wrapped_init(ServiceNode.java:139)
at com.my.platform.servicenode.core.ServiceNode.init(ServiceNode.java)
at com.my.platform.servicenode.core.Main.run(Main.java:47)
at com.my.platform.servicenode.core.Main.main(Main.java:86)

Seems to work fine under windows in Eclipse.

What am I missing?

David said...

Ok, I fixed that problem (it was my fault). I have another observation about this, however...

The code in the constructor didn't work for me unless I had the ClusterClassLoader implement NamedClassLoader, because the invoke call failed. The method registerGlobalLoader expectes a NamedClassLoader parameter apparently, and ClusterClassLoader does not implement it.

Putting the terracotta jar file on the classpath and implementing the interface made it work - however, this causes a build-time dependency on terracotta, and I don't want that.

I'm going to investigate doing some kind of runtime class manipulation to add the "implements" only if terracotta is detected, so that the code will work without terracotta as well.

Taylor said...

@David,

I don't think I quite understand, I thought I addressed that point in my blog. See the comment about using reflection in the constructor to detect if Terracotta is present.

Anonymous said...

There are ed hardy shirts
,pretty ed hardy shirt for men, ed hardy womens in the ed hardy online store designed by ed hardy ,many cheap ed hardy shirt ,glasses,caps,trouers ed hardy shirts on sale ,
You can go to edhardyshirts.com to have a look ,you may find one of ed hardy clothing fit for you
http://straighthairgirl.blog126.fc2.com
http://www.seriousblogging.com/crazygirlsshirts
http://www.free-blog-site.com/iammyself
http://d.hatena.ne.jp/hotfishing

puma mens shoes
nike air max ltd
NIKE air shoes

nike max ltd
orange converse
adidas shoes
nike shoes
puma shoes
addidas shoes
cheap converse shoes
cheap nike shoes
cheap puma shoes

Anonymous said...

cvjrt I like your blog. Thank you. They are really great . Ermunterung ++ .
Some new style Puma Speed is in fashion this year.
chaussure puma is Puma shoes in french . Many Franzose like seach “chaussure sport” by the internet when they need buy the Puma Shoes Or nike max shoes. The information age is really convenient .
By the way ,the nike max ltd is really good NIKE air shoes ,don’t forget buy the puma mens shoes and nike air max ltd by the internet when you need them . Do you know Nike Air Shoes is a best Air Shoes . another kinds of Nike shoes is better . For example , Nike Air Rift is good and Cheap Nike Shoes .the nike shox shoes is fitting to running.
Spring is coming, Do you think this season is not for Ugg Boots? maybe yes .but this season is best time that can buy the cheap ugg boots. Many sellers are selling discounted. Do not miss . Please view my fc2 blog and hair straighteners blog.
.thank you .

I like orange converse shoes ,I like to buy the cheap converse shoes by the internet shop . the puma shoes and the adidas shoes (or addidas shoes) are more on internet shop .i can buy the cheap nike shoes and cheap puma shoes online. It’s really convenient.
Many persons more like Puma basket shoes than nike air rift shoes . the Puma Cat shoes is a kind of Cheap Puma Shoes .
If you want to buy the Cheap Nike Air shoes ,you can buy them online. They are same as the Nike Air shoes authorized shop. Very high-caliber Air shoes and puma cat shoes . the cheap puma shoes as same as other.

Anonymous said...

ghd
cheap ghd
ghd straighteners
Benefit GHD
GHD IV Salon Styler
GHD MINI STYLER
GHD Precious gift
Gold GHD
Kiss GHD
New pink GHD
Pink GHD
Pure Black GHD
Pure White GHD
Purple GHD
Rare GHD
Babyliss
polo boots
polo shoes
cheap straighteners
ghd hair straighteners
hair straighteners
Purple GHD IV Styler
Pure White GHD
Pure Black GHD
Pink GHD IV Styler
Kiss GHD Styler
Gray GHD IV Styler
Gold GHD IV Styler
GHD Rare Styler
GHD IV Salon Styler
Black GHD IV Styler
Benefit GHD IV Styler

Unknown said...

i want to show you the following products:
gucci boots
gucci shoes
gucci t-shirt
gucci bags
gucci handbags
gucci clothing
gucci watch
gucci sunglasses
gucci jewelry
gucci boots
gucci shoes
gucci t-shirt
gucci bags
gucci handbags
gucci clothing
gucci watch
gucci sunglasses
gucci jewelry
gucci boots
gucci shoes
gucci t-shirt
gucci bags
gucci handbags
gucci clothing
gucci watch
gucci sunglasses
gucci jewelry

Unknown said...

ankh royalty

ankhroyalty

ankh royalty clothing

ankh royalty sweats

ankh royalty tracksuits
babyliss
babyliss pro
babyliss hair
babyliss i trim
babyliss flat iron
babyliss you curl
babyliss hair straighteners
babyliss hair straightener
babyliss straightener
babyliss straighteners
babyliss portability
babyliss straightening irons
babyliss hair iron
babyliss straightening iron

Unknown said...

Armani Sunglasses
Burberry Sunglasses
Cartier Sunglasses
Chanel Sunglasses
Coach Sunglasses
D&G Sunglasses
Dior Sunglasses
ED Hardy Sunglasses
Fendi Sunglasses
LV Sunglasses
Oakley Sunglasses
Police Sunglasses
Ray Ban Sunglasses
Roberto Cavalli Sunglasses
Versace Sunglasses

Unknown said...

air max,Nike air max
air max 90
nike air max 90
air max 1
air max 95
nike air max 360
nike air max ltd
air max ltd
air max 360
air max classic,Air Max Huarache
air max light
air max bw
air max plus
air max 180
air max nike
air max TN
air max 87
timberland
timberland boots
timberland shoes
cheap timberland boots
timberland uk
timberlands
timberland 2010
timberland outlet

Unknown said...

air jordan 2009
air jordan 2010
jumpman 23
jordan shoes outlet
jordan 6 rings
air jordan 10.5

shoes outlet
cheap shoes
AIR MAX
Bape Shoes
Nike Dunk
Nike Shox
Timberland Shoes
Prada Shoes
Gucci Shoes

cheap straightener
hair straightener
ghd outlet
ghd
2010 NEW GHD

Purple GHD IV Styler

Shareware zoo said...

Video-X-Ware.com offers you the best video converter reviews, video converter ratings, dvd ripper ratings to help you deal with your multimedia problems.
Video Converter
Mac Video Converter
DVD Ripper
DVD Creator
DVD Copy
iPhone Ringtone Converter
iPad Video Converter
iPod Video Converter
Blu-ray Ripper