Monday, March 31, 2008

Chronicles of a Terracotta Integration: Compass

Last week, I met up with Shay Banon, author of Compass, at the The Server Side: Vegas conference.  We thought it would be great to see if we couldn't crank out an integration between Terracotta and Compass.  You can read more about our integration from Shay himself.

I wanted to write a log of our efforts, because I thought it might provide some insight for anyone considering integrating Terracotta into their own project.  I was particularly happy with our effort, because it outlines what I feel is the best approach for developing with Terracotta.  The approach is actually quite simple.  Because Terracotta adds clustering concerns to your application using configuration, you don't write code directly to Terracotta.  Instead, you just write a simple POJO application without Terracotta, and then add the clustering later.  

So the approach I recommend is the following:

  1. Figure out how to implement the solution using a single JVM.  NO TERRACOTTA.  Use just simple POJOs and threads.

  2. Implement and test your solution.

  3. It helps to have envisioned, beforehand, what part of your implementation will become a Terracotta root.  But it's not necessary.  If your application is stateful, it will have a root.

  4. Using the root, start with a basic Terracotta config file, and build up the appropriate config file to cover all the instrumentation and locking.

  5. Test your application again, with a single jvm, but this time with Terracotta.

  6. Tune your implementation.

  7. Move to 2 or more JVMs.

That's it.  So how did this play out for the Compass integration?  Here is my rough recollection of the action.

10:00 am - Shook Hands - Shay and I met up at the conference.

10:05 am - Started coding - First we chatted a bit about our strategy.  It seemed easiest to start with the existing Lucene RAMDirectory implementation and tune it up a bit.

10:30 am - Strategy decided - Based on my knowledge of Terracotta, and Shay's knowledge of Lucene/Compass, we decided on the following:
  • Start with the Lucene RAMDirectory implementation, but rewrite it as necessary to fit a simple POJO model

  • Since RAMDirectory is mostly unmaintained, we knew we had to just go through the implementation and clean it up.  It comprises about 4 classes total, about 100 lines long, so the task was feasible.

  • Because Terracotta can just "plug" in to a well written application, and Shay has a comprehensive unit test suite (over 1,000 tests), a load test, and a concurrency test, we'd write the implementation first in POJOs

  • After verifying that the implementation works as expected in pure POJOs, then we would work on the configuration to inject Terracotta clustering

  • After running the solution with Terracotta, we would tune it.

  • And finally, we would wrap up various bits and pieces into a Terracotta Integration Module (TIM)
11:30 am - POJO Implementation done - We ended up rewriting the RAMDirectory, which was fine because it was in need of an overhaul anyway.  Rewriting its implementation meant we now had a good understanding of the implementation.
Just a quick note - it was a real joy coding with Shay.  He is a super smart guy, and it's great to work with someone like that.  Of note, he really understands synchronization, which is really important to write applications correctly.  Even better, he really got the principle of writing better code by writing less code.  We went through the RAMDirectory implementation with a weed wacker, and what was left was about 1/2 the code.  That was more readable and more maintainable.  And is better performing.  That was fun.
12:00 pm  - Unit Tests pass - With some minor corrections, we had unit tests passing.  We were both running out of power, and hungry, so we took a break to eat lunch, and agreed to resume in the afternoon.

1:30 pm - Write the Terracotta config file - While writing the POJO implementation, we already knew the key concepts we were going to need for writing up our config file.  We added the appropriate instrumentation.   We added the locking.   A few config statements later, we had a working Terracotta configuration.

2:00 pm - We had Compass running on Terracotta! - Approx. time elapsed?  2 1/2 hours (most of which was spent rewriting the RAMDirectory implementation)

2:30 pm - Tuning Time- At first Shay threw me - he said oh man it looks like it's running really fast.  Except it turns out he wasn't testing the right thing.  And then he tells me oh man its really slow!

Now don't misunderstand this.  I know Terracotta can go really fast.  But I wasn't in the least bit surprised.  And you shouldn't be either.  How many pieces of code have you ever written that compiled and ran correctly - on the first try?  Right.  One, if you are lucky.

Terracotta is kind of like that.  The first step is to get it right.  And that means synchronization, and locking, and once you have all that, your application runs correctly, but slowly.

Fortunately, it's easy to fix.

And so I taught Shay how to tune up his Terracotta integration.  Or rather, I showed him the tools he needed, and he went to town.  I just sort of stood by watching, giving the occasional comment or two.

This was the fun part.  It was time to take out the Admin console.  The Terracotta Admin console gives you a wealth of information about your application.  Of note:
  • You can browse your clustered data in realtime

  • You can monitor realtime statistics - including Terracotta txns/sec, Java Heap Memory, and CPU

  • You can access lock statistics using the lock profiler

  • You can snapshot over 30 metrics using the Statistics Recorder and visualize them using the Snapshot Visualizer
We started first with the object browser.  Once convinced that we had the right data in the cluster, we moved on to performance.

On our first run, we measured the Terracotta txns/sec.  I was actually pretty impressed to see that our server on his MacBook Pro was cranking out 10k/sec.  But I knew we wanted this number to be lower.  A lot lower.

So here comes the first rule for tuning Terracotta:  adjust your locking to match your workload.  It turns out that we had enabled an autolock for every single byte being written to the Lucene "files" - and this was hurting us pretty bad.  Because we already had a lock on our byte array that we were writing to, we actually just deleted the synchronization, and the lock config from the method that wrote bytes into the "file" - and we observed a big drop in the Terracotta txns/sec.  We went from the aforementioned 10k/sec to about 1750/sec.

Now what this means is that the Terracotta server was working just about 10x less for the same workload.  And that means we were doing more work/transaction, and so our performance improved accordingly.  You get the same effect with Hibernate - it batches up a bunch of little POJO updates into a single SQL statement - and that means you can do more real work because each SQL statement has more data in it.  Lots of little SQL statements means lots of overhead, and maybe more SQL queries executed/sec, but much less application txns/sec.  Same concept here with locking.

How did we identify what lock(s) to target?  

That's the second rule of tuning with Terracotta: USE THE ADMIN CONSOLE

We used the lock profiler feature included in the Admin Console to determine the exact stack trace that generated these locks. The process is simple:

  • Enable lock profiling with stack traces in the Admin Console, 

  • run your application,

  • then refresh the view to get a count of the lock acquires/releases/held times etc.

  • sort on # of lock acquires, and now you know what lock is being requested the most, what stack trace caused that lock, and what Terracotta config was responsible for making that lock.
Armed with this knowledge, Shay set about eliminating most of our superfluous locking.  Turns out that creating a Lucene "file" is a single threaded affair, so were able to create a single lock to cover the entire process of "writing" to a file, and that cut out about 90% of our locking.

At the end we got down to about 750 Terracotta txns/sec, which improved the application performance quite a bit.

Still not satisfied, we moved on to the Terracotta Statistics Recorder.  This is a new feature in Terracotta 2.6.

Turning on this feature records just about everything you ever wanted to know about your application, Terracotta, the JVM, and your system (including CPU, disk stats, and network stats).  You can export these stats as a CSV file, and import them into our Snapshot Visualizer Tool.  The SVT gives you a view like so:


4:30 pm - TIM time - We were pretty satisfied with the performance.  Even though we wanted more, Shay felt it was best to focus on turning Compass into a TIM (Terracotta Integration Module).

5:30 pm - Time to call it quits - We had hacked up the ant build.xml file to get ourselves a TIM in no-time - except that it wouldn't quite load correctly.  (Later we learned we had just specified the filename wrong - easy fix).

Overall, I thought we had a pretty good day.  We wrote and tuned a Terracotta integration in about 6 hours flat.   With a few more hours of work, Shay was able to complete the integration.

I was really happy to use some of the recent tools we have been building, like the Lock Profiler and the Statistics Recorder.  Seeing the real-world use of those was invaluable, and confirmed that our commitment to enabling the developer to self-tune by providing enhanced visibility is spot on.

I am looking forward to people downloading 2.6, trying out these awesome tools for themselves and providing feedback!

9 comments:

Stefan Fußenegger said...

Nice one!

I wonder, if it's possible to use this tuned version of Lucene's RAMDirectory without actually using Compass?

Taylor said...

Yes, it should be. I haven't figure out how to do that yet - Shay packaged the code into Compass, not Lucene.

While we could just fork the code and the config, I don't think that would be fair to the original intent.

Would love to hear ideas. Incidentally, I have heard anecdotally that some folks have done so and gotten huge improvements in speed over our existing RAMDirectrory integration module which is limited by the RAMDirectory implementation itself.

Stefan Fußenegger said...

Hi Taylor,

I already figured out how to do so. However, I only copied the code from Compass. I would love to help making this work as a tim for lucene 2.2 and hibernate search. I already contacted terracotta folks, but interest seems to be limited :(

Anonymous said...

(法新社a倫敦二B十WE四日電) 「情色二零零七」情趣產品大產自二十三日起在倫敦的成人網站肯辛頓奧林匹亞展覽館舉行,倫敦人擺脫對成人網站性的保守態度踴躍參觀,許多穿皮衣與塑膠緊身衣的好色之徒擠進這項世界規a片模最大的成人生活A片下載展,估計三天展期可吸引八萬多好奇民眾參觀。

活動計畫負責人米里根承諾:「要搞浪漫、誘惑人、玩虐待,你渴望的色情我們都有。」

他說:「時髦的設計與華麗女裝,從吊飾到束腹到真人大小的雕塑,是我們由今年展出的數千件情色產品精選出的一部分,參展產品還包括時尚服飾、貼身女用內在美、鞋子、珠寶、色情影片玩具、影片、藝術、圖書及遊戲,更不要說性成人電影av女優輔具及馬術裝備。」

參觀民眾遊覽a片兩百五十多個情色電影攤位情色電影,有性感服裝、玩具及情色食品,迎合各種品味。

大舞台上表演的是AV女優美國野蠻搖滾歌手瑪莉蓮曼情色森的前A片妻─全世界頭牌脫衣舞孃黛塔范提思,這是她今年在英國唯一一場表演。

以一九a片下載四零年代風格成人電影演出的黛塔色情范提思表演性感的天堂鳥、旋轉木馬及羽扇等舞蹈。

參展攤位有的av推廣情趣用品,有的AV成人影片開展示人成人影片體藝術和人體雕塑,也有情色藝術家工會成員提供建議。

Anonymous said...

03170
電報
op
iop
青島 温泉
uy
探偵
ent
you
qwo
wer
to
for
名刺
live
hi
sky
per
cou
xie
wei
shi
look
enw
up
here
go
hom
hous
gucl
shunk
cona
netbol
saish
town
yeawo

Anonymous said...

不動産
不動産投資
賃貸
新宿
不動産 東京
新宿 賃貸
現金化
マッサージ
温泉
価格
価格
価格比較
価格比較
名刺
名刺
名刺作成
名刺作成

Anonymous said...

fds
cen
don
kou
die
opin
idi
fds

コナン said...

ホームページ制作 仙台
任意売却
賃貸マンション 東京
賃貸マンション 札幌
仙台 中古車 販売
賃貸マンション 仙台
仙台 デリヘル 仙台 風俗
仙台 任意売却
ホームページ制作 横浜
横浜 手コキ
博多 デリヘル 求人 風俗

Shareware zoo said...

Video-X-Ware.com offers you the best video converter reviews, video converter ratings, dvd ripper reviews and dvd ripper ratings to help you deal with your multimedia problems.

http://www.video-x-ware.com

http://www.video-x-ware.com/ratings/video-converter/

http://www.video-x-ware.com

http://www.video-x-ware.com/ratings/video-converter/