Saturday, March 22, 2008

A Clustered ClassLoader

If you're building a distributed system, or contemplating building a distributed system, you might have run into this one before:


  1. You write and compile your classes in Eclipse
  2. You try out your classes on your laptop -- they work (woohoo!)
  3. Its a distributed system so you need to make sure your classes work in a true distributed environment
  4. You publish your classes to the distributed systems
  5. You try out your classes -- and they don't work (boo!)
  6. You fix the problem.
  7. You publish the classes again.
  8. Rinse, repeat.


After doing this a few dozen times, you find that publishing your classes to distributed systems is a total PITA that you would rather avoid altogether.

Or, you might have an application, like Master/Worker in which you deploy some part of the application at deploy time, but you deploy other parts of it during run-time. In the Master/Worker case, you deploy the Master and the Worker, but the Work comes and goes, and you'd like to be able to deploy new Work easily and trivially. In the Master/Worker case, since Masters are usually in control, and there is a farm of Workers, you'd like to deploy some new work to the Master, and let it send the Work to the Workers. Knowing about the Work up front on the Workers is a non-starter.

Some solutions to this problem?

  • Java has dynamic code loading capabilities already. Deploy your class files to a shared filesystem like NFS, and deploy your code to a shared directory.
  • Java also supports loading code from URLs (thanks to it's Applet heritage) so deploy your code to an HTTP server and you're set
  • Factor your application such that new classes aren't needed - just make the new definitions "data" driven
  • Embed a scripting engine, so you can pass Strings and interpret them as code - BeanShell, Jython, JRuby, Javascript, and Groovy all come to mind here...


Those are all fine solutions, but it never hurts to have more tools in your toolbox does it? Especially if you're already using Terracotta, wouldn't it be nice if there was some way to just leverage Terracotta's core clustering capabilities to build a clustered classloader?

I've done just that. Here's how it works:

  • Your application tries to instantiate a class, which means it asks the currently in scope ClassLoader to instantiate the class (by name)
  • By launching the application under the clustered classloader, it is in scope.
  • The clustered class loader has a Map<String, byte[]> that correlates classnames to bytes
  • The clustered class loader looks in this Map, if the classname is found, it uses the byte[] to create the requested class using defineClass()
  • If the class wasn't found in the Map, then it looks in the filesystem to find the class
  • If the class bytes are found on the filesystem, then it reads them into a byte[], and stashes them in the clustered Map<String, byte[]>
  • If the bytes aren't found, it just delegates to the parent classloader


I've omitted some of the finer details. The Map used is actually a Map<String, ClassMetaData> where ClassMetaData is a class that holds a long modified and byte[] bytes.

Let's have a look at the important parts of the ClusterClassLoader:

public class ClusterClassLoader extends ClassLoader
{
private static final String NAME = "ClusterClassLoader";

private static Map<String, Class> classes = new HashMap<String, Class>();
private static Map<String, ClassMetaData> bytes = new HashMap<String, ClassMetaData>();
private static transient boolean loaded;

...
ClusterClassLoader is defined to extend ClassLoader. It has a NAME field, which will be used to give a name to this classloader. This is a requirement for a classloader used by Terracotta. Normally Terracotta does this for you, but we are defining a new classloader, so we have to follow the naming rules for Terracotta (naming gives ClassLoaders across the cluster a unique identity).

A classes field is defined, which caches the result of the defineClass operation in the local JVM only. A bytes field is defined. This field is marked as a root, so that it can be shared with every other instance of ClusterClassLoader in the cluster.

The constructor detects if Terracotta is loaded using some reflection, and if so registers the classloader and sets a flag to enable cluster classloading features:
    public ClusterClassLoader()
{
super(ClusterClassLoader.class.getClassLoader());
try {
Class namedClassLoader = findClass("com.tc.object.loaders.NamedClassLoader");
Class helper = findClass("com.tc.object.bytecode.hook.impl.ClassProcessorHelper");
Method m = helper.getMethod("registerGlobalLoader", new Class[] { namedClassLoader });
m.invoke(null, new Object[] { this });
loaded = true;
} catch (Exception e) {
// tc is not present, so don't do anything fancy
loaded = false;
}
}


Next, the definition of loadClass is overridden:
    @Override
public Class<?> loadClass(String name) throws ClassNotFoundException
{
return findClass(name);
}
and so is findClass:
     @Override
protected Class<?> findClass(String name) throws ClassNotFoundException
{
if (!loaded) {
return getParent().loadClass(name);
}

Class result = null;
synchronized (classes) {
result = classes.get(name);
if (result != null) { return result; }

result = loadClassBytes(name);
if (result == null) { return getParent().loadClass(name); }
classes.put(name, result);
}

return result;
}
This is the bulk of the algorithm. The loaded flag is set when the class loader is instantiated. It used a bit of reflection to determine if Terracotta was even present in the JVM. If not, it is set to false, and the ClusterClassLoader just delegates to the parent class loader.

If Terracotta is present, then it checks to see if the class has already been defined. If so, it is returned directly from the classes cache. If it has not, then it gets the bytes from the loadClassBytes method. If that cannot find the bytes, then it asks the parent class loader to load the class.

The bulk of the implementation is done in the loadClassBytes method:

private Class loadClassBytes(String name) throws ClassNotFoundException
{
ClassMetaData metaData;

synchronized (bytes) {
try {
File f = null;
metaData = bytes.get(name);
URL resource = ClassLoader.getSystemResource(name.replace('.',File.separatorChar)+".class");
// if resource is non null, then the class is on the local fs (in the cp)
if (resource != null) {
f = new File(resource.getFile());
}

if (metaData != null) {
// if it's cached, but not on the fs, return it.
// if it's cached, but on the fs, check to see if it's
// up to date
if (f == null || metaData.modified >= f.lastModified()) {
return defineClass(name, metaData.bytes, 0, metaData.bytes.length, null);
}
}

// load from the fs
byte[] classBytes = loadClassData(f);
Class result = defineClass(name, classBytes, 0, classBytes.length, null);

try {
result.getDeclaredField("$__tc_MANAGED");
// it's managed so cache it
bytes.put(name, new ClassMetaData(f.lastModified(), classBytes));
} catch (NoSuchFieldException e) {
// not managed don't cache it
}
return result;
} catch (IOException e){
return null;
}
}
}
This method looks for the cached bytes, and for a file that corresponds to the class. If both are found, then it compares the modified date of the two. If the modified date of the bytes are greater than or equal to the file, then it returns the bytes in the cache. Otherwise it loads the bytes from the file. Once the bytes are loaded, defineClass is called to turn the bytes into a class file.

At this point, the ClusterClassLoader can check to see if the class is instrumented by Terracotta. Every instance of a class that is shared by Terracotta must be instrumented, so it's not necessary to cache class bytes for classes that are not instrumented by Terracotta. If the class is instrumented by Terracotta, then the ClusteredClassLoader stashes the bytes into the class bytes cache.

Click here if you would like to see the source code to ClusterClassLoader in its entirety

UPDATE: This project has been included in the tim-tclib project, and is a runnable sample. More details can be found in the sample readme.html

I've put the whole thing together as a simple runnable example. You just have to check out the source for the project, and run a few simple Maven commands. You can get the demo from here:

$ svn checkout http://svn.terracotta.org/svn/forge/projects/labs/tim-clusterclassloader clusterclassloader
$ cd clusterclassloader


The demo defines a main project, and two sub projects. The first sub project, sample, is responsible for putting classes into a queue. The second sub project, sample2, reads from the queue. To test the effectiveness of the cluster class loader, the second sample of course does not have the classes from the first sub project.

To run the demo:

  1. Build the project:

    $ mvn install
  2. Cd to the sample directory, compile and start a tc server:

    $ cd sample
    $ mvn package
    $ mvn tc:start
  3. Start the sample process:

    $ mvn tc:run
  4. In another terminal, cd to the sample2 directory:

    $ cd sample2
  5. Compile, and run the example:

    $ mvn package
    $ mvn tc:run



If you did everything correctly, you should see:

[INFO] [node] Waiting for work...
[INFO] [node] This is Callable2 calling!
In the second terminal (sample2). The message printed ("This is Callable2 calling!") is printed by a class that is only present in the classpath of the first instance (sample).

53 comments:

Sean said...

Hi, I've been trying out the example, but so far it isn't working for me.

I found that I had to check out clusteredclassloader from a different URL (which I basically guessed):
$ svn co http://svn.terracotta.org/svn/forge/projects/labs/tim-clusterclassloader clusteredclassloader

I also needed tim-parent:
$ svn co http://svn.terracotta.org/svn/forge/projects/tim-parent/tags/release-1.0.5 tim-parent
$ cd tim-parent; mvn install
$ cd ../clusteredclassloader
$ mvn install

But when I go to run the samples, the first program exits after a few seconds. Then the second program fails:

[INFO] [node] Exception in thread "main" com.tc.exception.TCClassNotFoundException: java.lang.ClassNotFoundException: org.terracotta.sample.
work.MyCallable2

Looks like the Worker is getting the job from the queue, but not the class.

What's wrong? Have I grabbed the wrong version from SVN?

Thanks,

Sean.

Taylor said...

@Sean,

Thanks, and good guesses! I forgot to run in a clean environment, so I missed a setting in my main pom.xml.

I have updated the main pom, so the tim-parent checkout and install shouldn't be necessary any longer.

I wiped my .m2 repo and started over again, following my instructions to the letter, and it works for me.

So I am not sure why it doesn't work for you - if you start over -- wipe your .m2/repository/org/terracotta directory -- and it still doesn't work feel free to jump on our forums or email me : tgautier -at- terracotta.org.

David said...

When I tried this technique for naming my classloader, I got the following running on Linux (jdk 1.6.0_05): java.lang.ClassNotFoundException: com.tc.object.loaders.NamedClassLoader
at java.lang.ClassLoader.findClass(ClassLoader.java:358)
at com.my.platform.servicenode.core.MyClassLoader.< init >(MyClassLoader.java:35)
at com.my.platform.servicenode.core.ServiceNode.loadLibraries(ServiceNode.java:185)
at com.my.platform.servicenode.core.ServiceNode.__tc_wrapped_init(ServiceNode.java:139)
at com.my.platform.servicenode.core.ServiceNode.init(ServiceNode.java)
at com.my.platform.servicenode.core.Main.run(Main.java:47)
at com.my.platform.servicenode.core.Main.main(Main.java:86)

Seems to work fine under windows in Eclipse.

What am I missing?

David said...

Ok, I fixed that problem (it was my fault). I have another observation about this, however...

The code in the constructor didn't work for me unless I had the ClusterClassLoader implement NamedClassLoader, because the invoke call failed. The method registerGlobalLoader expectes a NamedClassLoader parameter apparently, and ClusterClassLoader does not implement it.

Putting the terracotta jar file on the classpath and implementing the interface made it work - however, this causes a build-time dependency on terracotta, and I don't want that.

I'm going to investigate doing some kind of runtime class manipulation to add the "implements" only if terracotta is detected, so that the code will work without terracotta as well.

Taylor said...

@David,

I don't think I quite understand, I thought I addressed that point in my blog. See the comment about using reflection in the constructor to detect if Terracotta is present.

sexy said...

情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣,情趣,情趣,情趣,情趣,情趣,情趣,情趣,A片,視訊聊天室,聊天室,視訊,視訊聊天室,080苗栗人聊天室,上班族聊天室,成人聊天室,中部人聊天室,一夜情聊天室,情色聊天室,視訊交友網

免費A片,AV女優,美女視訊,情色交友,免費AV,色情網站,辣妹視訊,美女交友,色情影片,成人影片,成人網站,A片,H漫,18成人,成人圖片,成人漫畫,情色網,日本A片,免費A片下載,性愛

A片,色情,成人,做愛,情色文學,A片下載,色情遊戲,色情影片,色情聊天室,情色電影,免費視訊,免費視訊聊天,免費視訊聊天室,一葉情貼圖片區,情色,情色視訊,免費成人影片,視訊交友,視訊聊天,視訊聊天室,言情小說,愛情小說,AIO,AV片,A漫,avdvd,聊天室,自拍,情色論壇,視訊美女,AV成人網,色情A片,SEX,成人論壇

情趣用品,A片,免費A片,AV女優,美女視訊,情色交友,色情網站,免費AV,辣妹視訊,美女交友,色情影片,成人網站,H漫,18成人,成人圖片,成人漫畫,成人影片,情色網


情趣用品,A片,免費A片,日本A片,A片下載,線上A片,成人電影,嘟嘟成人網,成人,成人貼圖,成人交友,成人圖片,18成人,成人小說,成人圖片區,微風成人區,成人文章,成人影城,情色,情色貼圖,色情聊天室,情色視訊,情色文學,色情小說,情色小說,臺灣情色網,色情,情色電影,色情遊戲,嘟嘟情人色網,麗的色遊戲,情色論壇,色情網站,一葉情貼圖片區,做愛,性愛,美女視訊,辣妹視訊,視訊聊天室,視訊交友網,免費視訊聊天,美女交友,做愛影片

av,情趣用品,a片,成人電影,微風成人,嘟嘟成人網,成人,成人貼圖,成人交友,成人圖片,18成人,成人小說,成人圖片區,成人文章,成人影城,愛情公寓,情色,情色貼圖,色情聊天室,情色視訊,情色文學,色情小說,情色小說,色情,寄情築園小遊戲,情色電影,aio,av女優,AV,免費A片,日本a片,美女視訊,辣妹視訊,聊天室,美女交友,成人光碟

情趣用品.A片,情色,情色貼圖,色情聊天室,情色視訊,情色文學,色情小說,情色小說,色情,寄情築園小遊戲,情色電影,色情遊戲,色情網站,聊天室,ut聊天室,豆豆聊天室,美女視訊,辣妹視訊,視訊聊天室,視訊交友網,免費視訊聊天,免費A片,日本a片,a片下載,線上a片,av女優,av,成人電影,成人,成人貼圖,成人交友,成人圖片,18成人,成人小說,成人圖片區,成人文章,成人影城,成人網站,自拍,尋夢園聊天室

said...

runescape money runescape gold runescape money buy runescape gold buy runescape money runescape money runescape gold wow power leveling wow powerleveling Warcraft Power Leveling Warcraft PowerLeveling buy runescape gold buy runescape money runescape itemsrunescape accounts runescape gp dofus kamas buy dofus kamas Guild Wars Gold buy Guild Wars Gold lotro gold buy lotro gold lotro gold buy lotro gold lotro gold buy lotro gold runescape money runescape power leveling runescape money runescape gold dofus kamas cheap runescape money cheap runescape gold Hellgate Palladium Hellgate London Palladium Hellgate money Tabula Rasa gold tabula rasa money Tabula Rasa Credit Tabula Rasa Credits Hellgate gold Hellgate London gold wow power leveling wow powerleveling Warcraft PowerLeveling Warcraft Power Leveling World of Warcraft PowerLeveling World of Warcraft Power Leveling runescape power leveling runescape powerleveling eve isk eve online isk eve isk eve online isk tibia gold Fiesta Silver Fiesta Gold
Age of Conan Gold
buy Age of Conan Gold
aoc gold

呼吸机
无创呼吸机
家用呼吸机
呼吸机
家用呼吸机
美国呼吸机
篮球培训
篮球培训班
篮球夏令营
china tour
beijing tour
beijing travel
china tour
tibet tour
tibet travel
computer monitoring software
employee monitoring

said...

A片,A片,成人網站,成人漫畫,色情,情色網,情色,AV,AV女優,成人影城,成人,色情A片,日本AV,免費成人影片,成人影片,SEX,免費A片,A片下載,免費A片下載,做愛,情色A片,色情影片,H漫,A漫,18成人

a片,色情影片,情色電影,a片,色情,情色網,情色,av,av女優,成人影城,成人,色情a片,日本av,免費成人影片,成人影片,情色a片,sex,免費a片,a片下載,免費a片下載

情趣用品,情趣用品,情趣,情趣,情趣用品,情趣用品,情趣,情趣,情趣用品,情趣用品,情趣,情趣

A片,A片,A片下載,做愛,成人電影,.18成人,日本A片,情色小說,情色電影,成人影城,自拍,情色論壇,成人論壇,情色貼圖,情色,免費A片,成人,成人網站,成人圖片,AV女優,成人光碟,色情,色情影片,免費A片下載,SEX,AV,色情網站,本土自拍,性愛,成人影片,情色文學,成人文章,成人圖片區,成人貼圖

交友,AIO交友愛情館,AIO,成人交友,愛情公寓,做愛影片,做愛,性愛,微風成人區,微風成人,嘟嘟成人網,成人影片,成人,成人貼圖,18成人,成人圖片區,成人圖片,成人影城,成人小說,成人文章,成人網站,成人論壇,情色貼圖,色情貼圖,色情A片,A片,色情小說,情色小說,情色文學,寄情築園小遊戲, 情色A片,色情影片,AV女優,AV,A漫,免費A片,A片下載

kevio said...

威創牙醫診所除了提供優質的植牙技術外還提供假牙|矯正|牙周病治療,是值得您信賴的牙醫診所植牙權威。

獅王紋身工作室提供專業的無痛刺青技術,獅王紋身在世界TATTOO大賽中,獲獎無數,獅王紋身給您最時尚的作品。

陳駿逸皮膚科診所提供了治療痘痘的服務,皮膚雷射權威,包括雷射脈衝光除斑等,讓您回復青春蘋果臉。

ck皮件處理棧提供專業洗包包|洗鞋子|各式皮件修理保養疑難雜症都有服務,清洗包包專門店讓您的包包、鞋子、永遠保持最新的況態唷。

杏儒中醫診所提供了糖尿病的治療。

seo大師e王國幫您的網站輕鬆在您的行業裡站上第一頁,e王國的關鍵字行銷是您的好幫手,包括關鍵字自然排序、都能讓您獲得完美的效果,以目前的網路行銷不外乎是關鍵字自然排序為主、而關鍵字行銷seo又是e王國的強項也包括關鍵字廣告的服務,讓e王國幫您征服網海。

Weekfun星期樂優質生活平台-各式美食,精品,住宿,夜店,批發資訊,讓您不用出門也能盡知天下事,好的行銷帶你進墳場,好的行銷讓您上戰場,Weekfun星期樂"激動"行銷團隊將提供您前所未有的網路行銷,seo,關鍵字行銷,關鍵字廣告,關鍵字,自然排序,部落格行銷,等多項網路行銷類型供您挑選適合自已店裡的好行銷,網路爆紅不是夢!!

dsfgsdfgsdfgds said...

看房子,買房子,建商自售,自售,台北新成屋,台北豪宅,新成屋,豪宅,美髮儀器,美髮,儀器,髮型,EMBA,MBA,學位,EMBA,專業認證,認證課程,博士學位,DBA,PHD,在職進修,碩士學位,推廣教育,DBA,進修課程,碩士學位,網路廣告,關鍵字廣告,關鍵字,廣告,課程介紹,學分班,文憑,牛樟芝,段木,牛樟菇,日式料理, 台北居酒屋,燒肉,結婚,婚宴場地,推車飲茶,港式點心,尾牙春酒,台北住宿,國內訂房,台北HOTEL,台北婚宴,飯店優惠,台北結婚,婚宴場地,推車飲茶,港式點心,尾牙春酒,住宿,訂房,HOTEL,飯店,造型系列,學位,牛樟芝,腦磷脂,磷脂絲胺酸,SEO,婚宴,捷運,學區,美髮,儀器,髮型,牛樟芝,腦磷脂,磷脂絲胺酸,看房子,買房子,建商自售,自售,房子,捷運,學區,台北新成屋,台北豪宅,新成屋,豪宅,學位,碩士學位,進修,在職進修, 課程,教育,學位,證照,mba,文憑,學分班,網路廣告,關鍵字廣告,關鍵字,SEO,关键词,网络广告,关键词广告,SEO,关键词,网络广告,关键词广告,SEO,台北住宿,國內訂房,台北HOTEL,台北婚宴,飯店優惠,住宿,訂房,HOTEL,飯店,婚宴,台北住宿,國內訂房,台北HOTEL,台北婚宴,飯店優惠,住宿,訂房,HOTEL,飯店,婚宴,台北住宿,國內訂房,台北HOTEL,台北婚宴,飯店優惠,住宿,訂房,HOTEL,飯店,婚宴,結婚,婚宴場地,推車飲茶,港式點心,尾牙春酒,台北結婚,婚宴場地,推車飲茶,港式點心,尾牙春酒,結婚,婚宴場地,推車飲茶,港式點心,尾牙春酒,台北結婚,婚宴場地,推車飲茶,港式點心,尾牙春酒,結婚,婚宴場地,推車飲茶,港式點心,尾牙春酒,台北結婚,婚宴場地,推車飲茶,港式點心,尾牙春酒,居酒屋,燒烤,美髮,儀器,髮型,美髮,儀器,髮型,美髮,儀器,髮型,美髮,儀器,髮型,小套房,小套房,進修,在職進修,留學,證照,MBA,EMBA,留學,MBA,EMBA,留學,進修,在職進修,牛樟芝,段木,牛樟菇,住宿,民宿,飯宿,旅遊,住宿,民宿,飯宿,旅遊,住宿,民宿,飯宿,旅遊,住宿,民宿,飯宿,旅遊,住宿,民宿,飯宿,旅遊,住宿,民宿,飯宿,旅遊,住宿,民宿,飯宿,旅遊,美容,美髮,整形,造型,美容,美髮,整形,造型,美容,美髮,整形,造型,美容,美髮,整形,造型,美容,美髮,整形,造型,美容,美髮,整形,造型,美容,美髮,整形,造型,設計,室內設計,裝潢,房地產,設計,室內設計,裝潢,房地產,設計,室內設計,裝潢,房地產,設計,室內設計,裝潢,房地產,設計,室內設計,裝潢,房地產,設計,室內設計,裝潢,房地產,設計,室內設計,裝潢,房地產,設計,室內設計,裝潢,房地產,進修,在職進修,MBA,EMBA,進修,在職進修,MBA,EMBA,進修,在職進修,MBA,EMBA,進修,在職進修,MBA,EMBA,進修,在職進修,MBA,EMBA,進修,在職進修,MBA,EMBA,進修,在職進修,MBA,EMBA,住宿,民宿,飯店,旅遊,美容,美髮,整形,造型,設計,室內設計,裝潢,房地產,進修,在職進修,MBA,EMBA,關鍵字排名,網路行銷,关键词排名,网络营销,網路行銷,關鍵字排名,关键词排名,网络营销,羅志祥,周杰倫,五月天,蔡依林,林志玲,羅志祥,周杰倫,五月天,蔡依林,林志玲,PMP,在職專班,研究所在職專班,碩士在職專班,PMP,證照,在職專班,研究所在職專班,碩士在職專班,網頁設計,網站設計,網頁設計,網站設計,网页设计,网站设计,网站设计,网页设计

kiloi said...

nike tnEnter the necessary language
translation, up to 200 bytes winter, moves frequently in Chinanike chaussures showing that the deep strategy of the Chinese market. Harvard Business School, tn chaussures according to the relevant survey data show that in recent years the Chinese market three brands, Adidas, Li Ning market share at 21 percent, respectively,

kiloi said...

mens clothing men's sweate, cheap columbia jackets, lacoste sweater, ralph lauren polo shirts,ski clothing. Free Shipping, PayPal Payment. Enjoy your shopping experience on mensclothingstore.us

kiloi said...

cheap polos
polo shirts
ralph lauren polo shirtssport shoes
ugg boots
puma shoes
chaussures pumamp4
trade chinalacoste polo shirts
chaussure puma femmewedding dressestennis racket
cheap handbags
HAIR STRAIGHTENERS
ED HARDY SHIRTS

kiloi said...

HAIR STRAIGHTENERS
MENSCLOTHING mans clothing
cheap ugg boots
converse shoes
wedding dresses
wholesale polo shirts
brand clothingcheap clothing
clothes sportspolos shirtair shoesair shoesed hardy clothinged hardy clothing
英文推广

shen bing said...

cheap hair straightenerscheap flat ironnew polo shirtssexy lingerie storepolo shirtsnorth face jacketschi straightenerpink chichaussures puma chaussure puma

shen bing said...

hair straightenersugg bootscheap handbagscheap bagscheap pursetntattoo wholesalejackets worldjackets cartmen's clothingwomen's clothing

shen bing said...

handbags Louis Vuitton Vuitton handbags Balenciaga Balenciaga Bally handbagsBottega Veneta handbagsCartier handbagsChanel handbagsChloe handbagsChristian Dior handbagsCoach handbagsDolce Gabanna handbags

shen bing said...

Fendi handbags Givenchy handbags Gucci handbags Hermes handbags Jimmy Choo handbags Juicy Couture handbags lsabella Fiore handbags Miu Miu handbags Mulberry handbags Prada handbags Tods handbags Versace handbags Yves Saint Laurent handbags

shen bing said...

Men's Lacoste Polo Shirts Men's RL Striped Polo Shirts Women's Lacoste Polo Shirts Men's polo shirts Men's polo shirts Men's polo shirts 4 polo shirts Women's polo shirts 21 polo shirts Men's polo shirts Women's LACOSTE 5 PCS of Ralph Women's lacoste polo shirts

Talia.float said...

head junior tennis racketwilson tennis racquet
wilson tennis racket
head tennis racketbabolat tennis racket

Talia.float said...

Bought the Tennis Racquet is important, exercise can reduce the harm, especially for the wrist injury. But the good of Tennis Racket in general in the real prices are more expensive, so a lot of websites now have cheap tennis racquettennis racquet discountcheap tennis racketdiscount Tennis Racket .

ed-hardy-shirts said...

There are ed hardy shirts
,pretty ed hardy shirt for men, ed hardy womens in the ed hardy online store designed by ed hardy ,many cheap ed hardy shirt ,glasses,caps,trouers ed hardy shirts on sale ,
You can go to edhardyshirts.com to have a look ,you may find one of ed hardy clothing fit for you
http://straighthairgirl.blog126.fc2.com
http://www.seriousblogging.com/crazygirlsshirts
http://www.free-blog-site.com/iammyself
http://d.hatena.ne.jp/hotfishing

puma mens shoes
nike air max ltd
NIKE air shoes

nike max ltd
orange converse
adidas shoes
nike shoes
puma shoes
addidas shoes
cheap converse shoes
cheap nike shoes
cheap puma shoes

crystal said...

Thank you so much!!polo shirt men'ssweate,cheap polo shirts cheap columbia jackets, lacoste sweater, ralph lauren polo shirts,ski clothing. Free Shipping, PayPal Payment. Enjoy your shopping experience on mensclothingus.com。We have mens polo shirts.

crystal said...

Awesome!!!Best wishes for you !! wholesale polo shirts is the father of the summer should be prepared to most commonly used item, it has both style and shape of polo clothing, and vest with a random function, so that in the short-sleeved apply to both on many occasions, the pink and black color men's polo shirts brought into effect, lightweight cotton, linen texture to demonstrate masculine temperament and sense of fashion exhaustively. polo shirts for sale

crystal said...

Wonderful!!You can find the father who desire fashionable, intellectual cheap polos simultaneously, you can find a psychologist to study the most harmonious of families should be pink mens clothing, so do not want to take the mature route for the father, buy cheap polos, the learn from such a walk in between the formal and casual styling, refined style to create a sense of mild authority.

crystal said...

Have you ever wore Puma shoes, nike max shoes,nike shox shoes,Loose T-shirt or super-size clothing are the best partners, it allows you to exercise in relax. But that the classic style may not fit everyone. Short legs,big buttocks consciously out the best, you should never design yourself into a monster of no golden section point because beautiful shoes.Also,for ladies, in winter Ugg Boots are the excellent choice.

raul said...

cvjrt I like your blog. Thank you. They are really great . Ermunterung ++ .
Some new style Puma Speed is in fashion this year.
chaussure puma is Puma shoes in french . Many Franzose like seach “chaussure sport” by the internet when they need buy the Puma Shoes Or nike max shoes. The information age is really convenient .
By the way ,the nike max ltd is really good NIKE air shoes ,don’t forget buy the puma mens shoes and nike air max ltd by the internet when you need them . Do you know Nike Air Shoes is a best Air Shoes . another kinds of Nike shoes is better . For example , Nike Air Rift is good and Cheap Nike Shoes .the nike shox shoes is fitting to running.
Spring is coming, Do you think this season is not for Ugg Boots? maybe yes .but this season is best time that can buy the cheap ugg boots. Many sellers are selling discounted. Do not miss . Please view my fc2 blog and hair straighteners blog.
.thank you .

I like orange converse shoes ,I like to buy the cheap converse shoes by the internet shop . the puma shoes and the adidas shoes (or addidas shoes) are more on internet shop .i can buy the cheap nike shoes and cheap puma shoes online. It’s really convenient.
Many persons more like Puma basket shoes than nike air rift shoes . the Puma Cat shoes is a kind of Cheap Puma Shoes .
If you want to buy the Cheap Nike Air shoes ,you can buy them online. They are same as the Nike Air shoes authorized shop. Very high-caliber Air shoes and puma cat shoes . the cheap puma shoes as same as other.

haitao said...

cheap hair straighteners

new polo shirts
cheap Lacoste polo shirts
cheap Lacoste polo shirts

cheap handbags
cheap bags
puma chaussures
chaussures puma
chaussure puma

Men's North Face
Women's North Face


hair straighteners
sexy lingerie store
cheap ugg boots
tattoo wholesale
men's clothing
women's clothing


2009 nike shoes
new nike shoes
Women's max
Men's max 93
nike shox
Nike air force
Nike air max 2003
nike air max ltd
nike air max tn
Nike air rift
Nike air Yeezy
nike airmax
Nike air max 90
Nike air max 97
nike birds nest shoes
nike dunk
nike RT1 shoes
nike SB
nike shox shoes
Nike shox OZ shoes
Nike shox R2 shoes
Nike shox R3 shoes
Nike shox R4 shoes
Nike shox R5 shoes
Nike shox TL3
nike trainers lovers

tennis rackets
Wilson tennis rackets
HEAD tennis rackets
Babolat tennis rackets

lucyliu said...

nike air max 90
nike air max 95
nike air max tn
nike air rift
nike shox r4
nike air max 360
nike shox nz
puma mens shoes
puma shoes
puma speed
nike shoes
nike air
nike air shoes
puma cat
air max trainers
mens nike air max
nike shoes air max
nike shoes shox
air shoes
nike shoe cart
puma future
cheap puma
sports shoes
nike air rifts
nike air rift trainer
nike air
nike rift
nike rift shoes
cheap nike air rifts
bape shoes
jeans shop
diesel jeans
levis jeans

theprophet said...

sneakers shoes She continued,
"Why...? Don't you need someone to pose as your girlfriend this year?" Then he answered, "No, there is no need for that anymore......"
Before he can continue, he was interrupted, discount nike shoes"Oh yes! Must have found a girlfriend! nike shox r4 You haven't been searching for one for the past years, right?" The man looked up, as if he has struck gold, his face beamed and looked directly at the drunken girl. tn dollarHe replied, "Yes......you are right! I haven't been looking for anyone for the past years."
With that, the man darted across the floor and out the door, cheap nike shoesleaving the lady in much bewilderment. He finally realized that he has already found his dream girl, and she was.....the Vancouver girl all along! The drunken lady has said something that awoken him.
All along he has found his girl.nike tennis shoes That was why he did not bother to look further when he realized she was not coming back. It was not any specific girl he was seeking! cheap nike shoxIt was perfection that he wanted, and yes.....perfection!!
Relationship is something both parties should work on. Realizing that he had let away someone so important in his life, he decided to call her immediately. His whole mind was flooded with fear.free shipping shoes He was afraid that she might have found someone new or no longer had the same feelings anymore..... For once, he felt the fear of losing someone.
As it was Christmas eve, the line was quite hard to get through, especially an overseas call. He tried again and again, never giving up. Finally, he got through......precisely at 1200 midnight. He confessed his love for her and the girl was moved to tears. nike shoes It seemed that she never got over him! Even after so long, she was still waiting for him, never giving up.
He was so excited to meet her and to begin his new chapter of their lives. He decided to fly to Vancouver to join her. It was the happiest time of their lives! nike discount shoes But their happy time was short-lived. Two days before he was supposed to fly to Vancouver,cheap puma shoes he received a call from her father. She had a head-on car collision with a drunken driver. nike shox shoes She passed away after 6 hours in a coma.
The guy was devastated, as it was a complete loss. Why did fate played such cruel games with him? He cursed the heaven for taking her away from him, denying even one last look at her! How cruel he cursed! How he damned the Gods...!!nike free shoes How he hated himself....for taking so long to realize his mistake!! That was in 1996.
The moral of this story is :
Treasure what you have...
Time is too slow for those who wait;
Too swift for those who fear;
Too long for those who grief;
Too short for those who rejoice;
But for those who love...
Time is Eternity.
For all you out there with someone special in your heart, cherish that person, cherish every moment that you spend together that special someone, for in life, anything can happen anytime. buy shoes onlineYou may painfully regret, only to realise that it is too late.

xhy said...

I like your blog. Thank you. They are really great .
Some new style Puma Cat shoes is in fashion this year.
The Nike Air Shoes is a best Air Shoes .
Nike Air Rift is good and Cheap Nike Shoes.
If you are a fans of Puma basket,we would offer the good and Cheap Puma Shoes for you .the cheap ugg bootsis best christmas gift now.
The information age is really convenient .

LAN said...

puma mens shoes
puma shoes
puma speed
nike shoes
nike air
nike air shoes
nike air max 90
nike air max 95
nike air max tn
nike air rift
nike shox r4
nike air max 360
nike shox nz
puma cat
air max trainers
mens nike air max
sports shoes
nike air rifts
nike air rift trainer
nike air
nike shoes air max
nike shoes shox
air shoes
nike shoe cart
puma future
cheap puma
nike rift
jeans shop
diesel jeans
levis jeans
nike rift shoes
cheap nike air rifts
bape shoes

smallawei said...

性感睡衣,情趣睡衣,性感內衣褲,性感內衣,性感內褲,
性感貓裝,性感睡衣,貓裝,吊帶襪,情趣內褲,丁字褲,SM,

震動環,潤滑液,情趣禮物,情趣玩具,威而柔,精油,逼真按摩棒,數位按摩棒,


G點,按摩棒,轉珠按摩棒,變頻跳蛋,跳蛋,無線跳蛋,

飛機杯,男用強精長軟質套,男用強精短軟質套,充氣娃娃性感內褲,
自慰套,自慰套,情趣娃娃,自慰器,電動自慰器,充氣娃娃,
角色扮演,情趣,情趣用品,巴黎,
情趣用品,情趣,

mamamia said...

herve leger
herve leger dresses

chanel
chanel iman
chanel outlet
chanel handbag
Chanel Wallet
Chanel Watch
chanel purse

Chanel Sunglasses
chanel bags
chanel earrings
chanel jewelry
chanel shoes

mamamia said...

ghd
cheap ghd
ghd straighteners
Benefit GHD
GHD IV Salon Styler
GHD MINI STYLER
GHD Precious gift
Gold GHD
Kiss GHD
New pink GHD
Pink GHD
Pure Black GHD
Pure White GHD
Purple GHD
Rare GHD
Babyliss
polo boots
polo shoes
cheap straighteners
ghd hair straighteners
hair straighteners
Purple GHD IV Styler
Pure White GHD
Pure Black GHD
Pink GHD IV Styler
Kiss GHD Styler
Gray GHD IV Styler
Gold GHD IV Styler
GHD Rare Styler
GHD IV Salon Styler
Black GHD IV Styler
Benefit GHD IV Styler

my blog said...

i want to show you the following products:
gucci boots
gucci shoes
gucci t-shirt
gucci bags
gucci handbags
gucci clothing
gucci watch
gucci sunglasses
gucci jewelry
gucci boots
gucci shoes
gucci t-shirt
gucci bags
gucci handbags
gucci clothing
gucci watch
gucci sunglasses
gucci jewelry
gucci boots
gucci shoes
gucci t-shirt
gucci bags
gucci handbags
gucci clothing
gucci watch
gucci sunglasses
gucci jewelry

xiansheng said...

rain boot
rainboots
rainboot
rain boots
rain coat
rain coat
raincoat
raincoats
rain coats
rain wear

abercrombie and fitch shirts
abercrombie outlet
abercrombie fitch outlet
abercrombie & fitch clothing
abercrombie and fitch clothes
abercrombie and fitch clothing
abercrombie fitch
abercrombie fitch clothing
abercrombie & fitch
abercrombie and fitch
abercrombie and fitch outlet
cheap abercrombie fitch
abercrombie shirtabercrombie fitch jacketabercrombie jacketabercrombie and fitch hoodiet

xiansheng said...

ankh royalty

ankhroyalty

ankh royalty clothing

ankh royalty sweats

ankh royalty tracksuits
babyliss
babyliss pro
babyliss hair
babyliss i trim
babyliss flat iron
babyliss you curl
babyliss hair straighteners
babyliss hair straightener
babyliss straightener
babyliss straighteners
babyliss portability
babyliss straightening irons
babyliss hair iron
babyliss straightening iron

Linn said...

adidas shoes
adidas shoe
adidas basketball shoes
adidas running shoes
adidas tennis shoes
adidas sneakers

Linn said...

adidas shoes
adidas shoe
adidas basketball shoes
adidas running shoes
adidas tennis shoes
adidas sneakers

Linn said...

jordan shoes
cheap jordan shoes
jordan 1
jordan 2
jordan 3
jordan 4
jordan 5
jordan 6
jordan 7
jordan 8
jordan 9
jordan 10
jordan 11
jordan 12
jordan 13
jordan 14
jordan 15
jordan 16
jordan 17
jordan 18
jordan 19
jordan 20
jordan 21
jordan 22
jordan 23
Michael Jackson memorial
air jordan 2009
air jordan 2010
jumpman 23

Hanthan said...

Armani Sunglasses
Burberry Sunglasses
Cartier Sunglasses
Chanel Sunglasses
Coach Sunglasses
D&G Sunglasses
Dior Sunglasses
ED Hardy Sunglasses
Fendi Sunglasses
LV Sunglasses
Oakley Sunglasses
Police Sunglasses
Ray Ban Sunglasses
Roberto Cavalli Sunglasses
Versace Sunglasses

Hanthan said...

air max,Nike air max
air max 90
nike air max 90
air max 1
air max 95
nike air max 360
nike air max ltd
air max ltd
air max 360
air max classic,Air Max Huarache
air max light
air max bw
air max plus
air max 180
air max nike
air max TN
air max 87
timberland
timberland boots
timberland shoes
cheap timberland boots
timberland uk
timberlands
timberland 2010
timberland outlet

cainiao1234 said...

air jordan 2009
air jordan 2010
jumpman 23
jordan shoes outlet
jordan 6 rings
air jordan 10.5

shoes outlet
cheap shoes
AIR MAX
Bape Shoes
Nike Dunk
Nike Shox
Timberland Shoes
Prada Shoes
Gucci Shoes

cheap straightener
hair straightener
ghd outlet
ghd
2010 NEW GHD

Purple GHD IV Styler

Michael Jacksson said...

buy viagra
viagra online
generic viagra

jimmychooshoes said...

Jimmy Choo Embossed leather sandals
Jimmy Choo Zero metallic-leather sandals
Christian Louboutin Alta Perla Strass 100 sandals
mcqueen shoes
alexander mcqueen heels
Alexander McQueen Store
Jimmy Choo sale
Replica jimmy choo shoes
manolo blahnik online store
manolo shoes
manolo blahnik sale
replica christian louboutin shoes
discount christian louboutin shoes
alexander mcqueen shoes
jimmy choo outlet
jimmy choos
jimmy shoes
Manolo Blahnik Shoes
christian shoes

britney sayers said...

http://www.fishingcostaricaexperts.com/
 
Costa rica Fishing

2010 Wedding Dresses said...

thanks for your sharing! great helpful!!!Thank you
Louboutin Shoes 
Christian Louboutin Shoes
Christian Louboutin Pumps 
louboutin
Christian Louboutin Boots 
Christian Louboutin Pumps
Christian Louboutin Heels 
Christian Louboutin Sandal
Pigalle Christian Louboutin 
fur boots
christian louboutin fur boots
louboutin pumps
louboutin fur boots
coyote fur boots
christian louboutin heels
silver strappy sandals
christian louboutin toundra
loubitan studded bow peep
Wedding Dress Shops
vintage wedding dresses 
wedding dresses
cheap mobile phone
Wedding Dress Shops
Wedding Dresses 2011
Wedding Dress Shops
Off the Shoulder Wedding Dresses
bridal jackets
wedding jackets
bridal jackets
halter neck wedding dresses
off the shoulder wedding dresses
vintage wedding dresses
wedding dresses with sleeves
wedding jackets for the bride

Friday said...

The louboutin heels is designed for the ladies. Since the pumps was born, the ladies life become colorful. The christian louboutin thong sandals are the god's masterwork. Who invited thechristian louboutin pumps? Seldom people knew, but I think every lady would be grateful for him. Among the countless pumps, the christian louboutin platforms is the most outstanding ones. The elegant pattern, the delicate style all mold the ladies perfect leg profile. Flowers in the spring of 2011 creeping, up from hair to clothes continue to footwear, have had a brilliant up. In such a glamor, spring and summer flowers now here. Romance is a woman's mood, exquisite flowers just right of expression in our gestures, the woman, how can we not love the romantic temperament so that they distributed the flowers do? 2011 flowers bloom will enjoy different poses!The christian louboutin wedges also can adds the hright of the ladies, it bring surprise to the short lady. Especially the red sole of the louboutin heels, magic and sexy, many ladies are crazy. The red sole, the first feature of the Christian Louboutin stroe.

Olivia said...

Your Escort Agency offers exclusive and most beautiful London escort girls of various nationalities.
Your Escort Agency

Shareware zoo said...

Video-X-Ware.com offers you the best video converter reviews, video converter ratings, dvd ripper ratings to help you deal with your multimedia problems.
Video Converter
Mac Video Converter
DVD Ripper
DVD Creator
DVD Copy
iPhone Ringtone Converter
iPad Video Converter
iPod Video Converter
Blu-ray Ripper

London Escort Agency said...

Bestescort4U more then ten years providing best London escorts companionship in the UK.
London escorts

Escorts London said...

Hot - Collection is a honest and confidential London escort agency which provides genuine London escorts girls for gentlemen of taste.
London escorts