tag:blogger.com,1999:blog-350589372024-03-07T13:37:42.573-08:00java.think()Thoughts on Java, Design, Patterns, and StrategiesTaylorhttp://www.blogger.com/profile/07193759050963768511noreply@blogger.comBlogger48125tag:blogger.com,1999:blog-35058937.post-20552717459413863132010-06-13T13:01:00.000-07:002010-06-14T09:22:38.482-07:00Configuring a Grails App for Clustering using EhcacheClustering Grails using Ehcache is very easy. Here is how to do it in just a few simple steps.<br /><br />Requirements:<br />1) Grails 1.3.1 installed. <a href="http://www.grails.org/Download">Download Grails here</a> if you don't already have it installed.<br />2) Terracotta 3.2.1 installed. <a href="http://www.terracotta.org/dl/">Download Terracotta here</a> if you don't already it installed.<br /><br />This post assumes you have Grails installed to $GRAILS_HOME and Terracotta installed to $TERRACOTTA_HOME.<br /> <br />Before setting up your application for clustering, we'll need a Grails app. If you don't already have a Grails app, let's create one. We just repeat the steps listed on the <a href="http://www.grails.org/Quick+Start">Grails Quick Start Page</a>. I will create a simple application called "Events" which create and stores events:<br /><br /><b>Step 1. Create the application</b><br /><br /><code>$ grails create-app events</code><br /><br /><b>Step 2. Create a domain class</b><br /><br /><code>grails create-domain-class Event</code><br /><br />Edit the generated Event domain class <code>grails-app/domain/events/Event.groovy</code> and add some fields to it:<br /><br /><pre name="code" class="java">package events<br />class Event {<br /> Date date<br /> String title<br />}<br /></pre><br /><b>Step 3. Create a controller</b><br /><br /><code>$ grails create-controller Event</code><br /><br />Edit the grails-app/controllers/events/EventController.groovy to implement default scaffolding:<br /><br /><pre name="code" class="java">package events<br /><br />class EventController {<br /> def scaffold = Event<br />}<br /></pre><br /><b>Step 4. Run the app</b><br /><br /><code>$ grails run-app</code><br /><br />And browse to <a href="http://localhost:8080/events/event">http://localhost:8080/events/event</a><br /><br />Now we have a complete Grails application. Let's add Terracotta:<br /><br /><b>Step 5. Configure your domain class for caching</b><br /><br />You will need to tell Hibernate that your domain class is cacheable. Edit the domain class at <code>grails-app/domain/events/Event.groovy</code> and add the cache directive:<br /><br /><pre name="code" class="java">package events<br /><br />class Event {<br /> static mapping = {<br /> cache true<br /> }<br /><br /> Date date<br /> String title<br />}<br /></pre><br /><b>Step 6. Configure Grails to use the latest version of Ehcache with Terracotta support built-in</b><br /><br />Edit the config file at <code>grails-app/conf/BuildConfig.groovy</code>. Update the section which imports the default global settings like so to update the depencencies to the latest version of Ehcache (sidenote, Ehcache version 2.1.0 depends on Terracotta version 3.2.1, don't get confused by the version numbers - they don't line up because the two products are different even if owned by the same company): <br /><br /><pre name="code" class="java">grails.project.dependency.resolution = {<br /> // inherit Grails' default dependencies<br /> inherits( "global" ) {<br /> // uncomment to disable ehcache<br /> // excludes 'ehcache'<br /> runtime 'net.sf.ehcache:ehcache-core:2.1.0'<br /> runtime 'net.sf.ehcache:ehcache-terracotta:2.1.0'<br /> }<br /><br /> <rest of file here><br /></pre><br /><b>Step 7. Configure Ehcache to use Terracotta.</b><br /><br />By default Ehcache caches are not configured for Terracotta support. Enable this by overriding the built-in Ehcache defaults by adding the file <code>grails-app/conf/ehcache.xml</code> with the following contents:<br /><br /><pre name="code" class="xml"><ehcache name="EventCache"><br /> <defaultCache<br /> maxElementsInMemory="10"<br /> eternal="false"<br /> timeToIdleSeconds="120"<br /> timeToLiveSeconds="120"<br /> overflowToDisk="false"><br /> <terracotta/><br /> </defaultCache><br /> &;t'terracottaConfig url="localhost:9510"/><br /></ehcache><br /></pre><br /><b>Step 8. Start a Terracotta Server</b><br /><br />Terracotta requires that a Server is running. Start it now:<br /><br /><code>$ $TERRACOTTA_HOME/bin/start-tc-server.sh</code><br /><br /><b>Step 9. Start a developer console</b><br /><br />To observe caching in action, start a Terracotta Developer Console:<br /><br /><code>$ $TERRACOTTA_HOME/bin/dev-console.sh</code><br /><br /><b>Step 10. Run the app again</b><br /><br /><code>$ grails run-app</code><br /><br />You can monitor the cache stats in the Terracotta Developer Console. To do so, <b>make sure you turn on statistics gathering:</b><br /><ol><li>Click on Ehcache</li><li>Click on Statistics</li><li>Find the "Enable Statistics" button and click it</li></ol><br />The following screen shot shows where to click:<br /><br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhn2krAk0nhYFuy_PHJGBqZ72XIwtDj-8b6IvzlePa3gAnVRBn1AejeuAmKH3Z5tCUo1PdM6YVHSLNTG6nFLMJgVy0lnZItbOsLfXDmlygwVEcYN0ryu9GKf-ciCnQNper8NvgE/s1600/Screen+shot+2010-06-13+at+1.31.49+PM.jpg"><img style="cursor:pointer; cursor:hand;width: 400px; height: 253px;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhn2krAk0nhYFuy_PHJGBqZ72XIwtDj-8b6IvzlePa3gAnVRBn1AejeuAmKH3Z5tCUo1PdM6YVHSLNTG6nFLMJgVy0lnZItbOsLfXDmlygwVEcYN0ryu9GKf-ciCnQNper8NvgE/s400/Screen+shot+2010-06-13+at+1.31.49+PM.jpg" border="0" alt=""id="BLOGGER_PHOTO_ID_5482360913635229522" /></a><br /><br />Now, navigate to the application at <a href="http://localhost:8080/events/event">http://localhost:8080/events/event</a>. Create an event. After creating the event, you are left on a page that views the event. Press "Refresh" on your browser a few times, and notice that you get activity in the Developer Console Statistics window.<br /><br />Here's what it should look like:<br /><br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhzSP0z35hchzoZYuDaofigrLwKDuTiFQ_hJIKe6Kd8FkGPOC4WaMiqLmzK7_bSgm1_RyW6wZC8C8Uy8m9mBli_XLin9fo5sT_ZaHvUBuks8klQXbVrpgZQwDlIpy8xdbVFPLck/s1600/Screen+shot+2010-06-13+at+1.32.56+PM.jpg"><img style="cursor:pointer; cursor:hand;width: 400px; height: 253px;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhzSP0z35hchzoZYuDaofigrLwKDuTiFQ_hJIKe6Kd8FkGPOC4WaMiqLmzK7_bSgm1_RyW6wZC8C8Uy8m9mBli_XLin9fo5sT_ZaHvUBuks8klQXbVrpgZQwDlIpy8xdbVFPLck/s400/Screen+shot+2010-06-13+at+1.32.56+PM.jpg" border="0" alt=""id="BLOGGER_PHOTO_ID_5482360516824243202" /></a><br /><br />That's it - have fun with clustered Ehcache for Grails!Taylorhttp://www.blogger.com/profile/07193759050963768511noreply@blogger.com5tag:blogger.com,1999:blog-35058937.post-535631590922089722010-01-24T12:31:00.000-08:002010-08-06T14:46:45.383-07:00Tutorial - Integrating Terracotta EHCache for Hibernate with Spring PetClinic<span style="font-weight:bold;">Now updated for Terracotta 3.3!</span><br /><br />With Terracotta's latest 3.2 release, configuring 2nd Level Cache for Hibernate is incredibly simple. What's more is that using the included Hibernate console, we can identify the hot spots in our application, and eliminate unwanted database activity.<br /><br />In this blog I will show you how to setup and install Terracotta EHCache for Hibernate into the venerable Spring PetClinic application. After that we will identify where database reads can be converted to cache reads. By the end of the tutorial, we will convert all database reads into cache reads, demonstrating 100% offload of the database.<br /><br />What you'll need:<br /><ul><li>Java 1.5 or greater</li><li>Ant</li><li>Tomcat</li></ul>Let's get started.<br /><h4>Step 1 - Download and unzip the Spring PetClinic Application</h4>Go to the <a href="http://www.springsource.org/download" title="Spring download site">Spring download site</a> and download Spring 2.5.6 SEC01 with dependencies. Of course you can download other versions, but 2.5.6 SEC01 is the version I used for this tutorial.<br /><br />Unzip the installation into $SPRING_HOME.<br /><h4>Step 2 - Build PetClinic for use with Hibernate</h4>The PetClinic application is location in $SPRING_HOME/samples/petclinic. Cd to this directory.<br /><br />The first thing we need to do is setup PetClinic to use Hibernate. To do so, update the web.xml file located in src/war/WEB-INF/web.xml. Locate the section called contextConfigLocation and update it to look like the following (comment out the JDBC config file and un-comment out the hibernate config file):<br /><pre name="code" class="xml"><context-param><br /> <param-name>contextConfigLocation</param-name><br /> <param-value>/WEB-INF/applicationContext-hibernate.xml</param-value><br /> <!-- <param-value>/WEB-INF/applicationContext-jdbc.xml</param-value><br /> <param-value>/WEB-INF/applicationContext-jpa.xml</param-value< --><br /> <!-- To use the JPA variant above, you will need to enable Spring load-time<br /> weaving in your server environment. See PetClinic's readme and/or <br /> Spring's JPA documentation for information on how to do this. --><br /> </context-param><br /></pre>Now, build the application:<br /><blockquote style="padding-top: 10px; padding-right: 10px; padding-bottom: 10px; padding-left: 10px; border-top-width: 1px; border-right-width: 1px; border-bottom-width: 1px; border-left-width: 1px; border-top-color: rgb(221, 221, 221); border-right-color: rgb(221, 221, 221); border-bottom-color: rgb(221, 221, 221); border-left-color: rgb(221, 221, 221); border-top-style: dashed; border-right-style: dashed; border-bottom-style: dashed; border-left-style: dashed; ">$ ant warfile<br />...<br />BUILD SUCCESSFUL<br />Total time: 20 seconds<br /></blockquote><h4>Step 3 - Start PetClinic</h4>First, start the HSQLDB database:<br /><blockquote style="padding-top: 10px; padding-right: 10px; padding-bottom: 10px; padding-left: 10px; border-top-width: 1px; border-right-width: 1px; border-bottom-width: 1px; border-left-width: 1px; border-top-color: rgb(221, 221, 221); border-right-color: rgb(221, 221, 221); border-bottom-color: rgb(221, 221, 221); border-left-color: rgb(221, 221, 221); border-top-style: dashed; border-right-style: dashed; border-bottom-style: dashed; border-left-style: dashed; "><span style="font-family:'Courier New';">$ cd db/hsqldb<br />$ ./server.sh</span></blockquote>Next, copy the WAR file to your Tomcat's webapps directory:<br /><blockquote style="padding-top: 10px; padding-right: 10px; padding-bottom: 10px; padding-left: 10px; border-top-width: 1px; border-right-width: 1px; border-bottom-width: 1px; border-left-width: 1px; border-top-color: rgb(221, 221, 221); border-right-color: rgb(221, 221, 221); border-bottom-color: rgb(221, 221, 221); border-left-color: rgb(221, 221, 221); border-top-style: dashed; border-right-style: dashed; border-bottom-style: dashed; border-left-style: dashed; "><span style="font-family:'Courier New';">$ cp dist/petclinic.war $TOMCAT_HOME/webapps</span></blockquote>And start Tomcat:<br /><blockquote style="padding-top: 10px; padding-right: 10px; padding-bottom: 10px; padding-left: 10px; border-top-width: 1px; border-right-width: 1px; border-bottom-width: 1px; border-left-width: 1px; border-top-color: rgb(221, 221, 221); border-right-color: rgb(221, 221, 221); border-bottom-color: rgb(221, 221, 221); border-left-color: rgb(221, 221, 221); border-top-style: dashed; border-right-style: dashed; border-bottom-style: dashed; border-left-style: dashed; "><span class="Apple-style-span" style="font-family:'Courier New';">$ $TOMCAT_HOME/bin/catalina.sh start<br />...</span></blockquote><br />You should now be able to access PetClinic at <a href="http://localhost:8080/petclinic">http://localhost:8080/petclinic</a> and see the home screen:<br /><br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgrLGZgRdiLp8InJ_v5Q2As07sY8bMB35stxnuI8uE4HT3i-JGF21EclZ15Id7KDxoCJwjg6hG8fVQxe-z8WkYdj4XN3Pt8j2t57RRxpTvjvYK1eUJb6IHXug4nY3ypob7w6c0Q/s1600-h/Screen+shot+2010-01-24+at+11.06.05+AM.png"><img style="cursor:pointer; cursor:hand;width: 200px; height: 94px;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgrLGZgRdiLp8InJ_v5Q2As07sY8bMB35stxnuI8uE4HT3i-JGF21EclZ15Id7KDxoCJwjg6hG8fVQxe-z8WkYdj4XN3Pt8j2t57RRxpTvjvYK1eUJb6IHXug4nY3ypob7w6c0Q/s200/Screen+shot+2010-01-24+at+11.06.05+AM.png" border="0" alt=""id="BLOGGER_PHOTO_ID_5430425459472652434" /></a><br /><h4>Step 4 - Install Terracotta</h4>Download Terracotta from the <a href="http://www.terracotta.org/download" title="http://www.terracotta.org/download">http://www.terracotta.org/download</a>. Unzip and install into $TC_HOME<br /><h4>Step 5 - Configure Spring PetClinic Application to use Ehcache as a Hibernate Second Level Cache</h4>First, the Terracotta Ehcache libraries must be copied to the WEB-INF/lib directory so they can be compiled into the PetClinic WAR file. The easiest way to do this is to update the build.xml file that comes with Spring PetClinic. Add two properties to the top of the build file (make sure to replace PATH_TO_YOUR_TERRACOTTA with your actual path):<br /><pre name="code" class="xml"><property name="tc.home" value="PATH_TO_YOUR_TERRACOTTA" /><br /><property name="ehcache.lib" value="${tc.home}/ehcache/lib" /></pre>Then, update the lib files section - locate the section that starts with the comment "copy Tomcat META-INF", adding the Terracotta hibernate jars like so:<br /><pre name="code" class="xml"><!-- copy Tomcat META-INF --><br /><copy todir="${weblib.dir}" preservelastmodified="true"><br /> <fileset dir="${tc.home}/lib"><br /> <include name="terracotta-toolkit*.jar"><br /> </include><br /> </fileset><br /> <fileset dir="${ehcache.lib}"><br /> <include name="ehcache*.jar"><br /> </include><br /> </fileset><br /> ...<br /></copy></pre>The Sprint PetClinic application by default includes Ehcache libraries, but we are setting up the application to use the latest version of Ehcache. Therefore you should also remove the sections in this file that copy the ehcache libraries to the WAR file:<br /><pre name="code" class="xml"><!--<br /> <fileset dir="${spring.root}/lib/ehcache"><br /> <include name="ehcache*.jar"/><br /> </fileset><br /> --><br /></pre><i>Note: there are two such entries, make sure to update them both!</i><br /><br />You'll also need to update the applicationContext.xml file, located in war/WEB-INF/applicationContext-hibernate.xml. This is the Spring configuration file, and it contains the properties that configure Hibernate. We need to update the 2nd level cache provider settings so Hibernate will use Terracotta. Update the HibernateSessionFactory like so:<br /><pre name="code" class="xml"><!-- Hibernate SessionFactory --><br /> <bean id="sessionFactory" class="org.springframework.orm.hibernate3.LocalSessionFactoryBean" ref="dataSource" mappingresources="petclinic.hbm.xml"><br /> <property name="hibernateProperties"><br /> <props><br /> <prop key="hibernate.dialect">${hibernate.dialect}</prop><br /> <prop key="hibernate.show_sql">${hibernate.show_sql}</prop><br /> <prop key="hibernate.generate_statistics">${hibernate.generate_statistics}</prop><br /> <prop key="hibernate.cache.use_second_level_cache">true</prop><br /> <prop key="hibernate.cache.region.factory_class">net.sf.ehcache.hibernate.EhCacheRegionFactory</prop><br /> </props><br /> </property><br /> <property name="eventListeners"><br /> <map><br /> <entry key="merge"><br /> <bean class="org.springframework.orm.hibernate3.support.IdTransferringMergeEventListener"><br /> </bean></entry><br /> </map><br /> </property><br /> </bean></pre>As written, the Spring PetClinic application entities are not configured for caching. Each entity configuration in Hibernate requires caching to be explicitly enabled before caching is available. To enable caching, open the petclinic.hbm.xml file located in src/petclinic.hbm.xml and add caching entries to each entity definition. Here are the entity definitions I used:<br /><pre name="code" class="xml"><class name="org.springframework.samples.petclinic.Vet" table="vets"><br /> <cache usage="read-write"/><br /> ....<br /></class><br /><class name="org.springframework.samples.petclinic.Specialty" table="specialties"><br /> <cache usage="read-only"/><br /> ....<br /></class><br /><class name="org.springframework.samples.petclinic.Owner" table="owners"><br /> <cache usage="read-write"/><br /> ....<br /></class><br /><class name="org.springframework.samples.petclinic.Pet" table="pets"><br /> <cache usage="read-write"/><br /> ....<br /></class><br /><class name="org.springframework.samples.petclinic.PetType" table="types"><br /> <cache usage="read-only"/><br /> ....<br /></class><br /><class name="org.springframework.samples.petclinic.Visit" table="visits"><br /> <cache usage="read-write"/><br /> ....<br /></class></pre><h4>Step 6 - Rebuild and re-deploy</h4>Now that the PetClinic app is configured for use with Terracotta and Hibernate 2nd level cache, re-build the war file and re-deploy it to your tomcat installation:<blockquote style="padding-top: 10px; padding-right: 10px; padding-bottom: 10px; padding-left: 10px; border-top-width: 1px; border-right-width: 1px; border-bottom-width: 1px; border-left-width: 1px; border-top-color: rgb(221, 221, 221); border-right-color: rgb(221, 221, 221); border-bottom-color: rgb(221, 221, 221); border-left-color: rgb(221, 221, 221); border-top-style: dashed; border-right-style: dashed; border-bottom-style: dashed; border-left-style: dashed; "><span class="Apple-style-span" style="font-family:'Courier New';">$ ant warfile<br />...<br />BUILD SUCCESSFUL<br />Total time: 20 seconds<br />$ $TOMCAT_HOME/bin/catalina.sh stop<br />$ rm -rf $TOMCAT_HOME/webapps/petclinic<br />$ cp dist/petclinic.war $TOMCAT_HOME/webapps<br /></span></blockquote><br /><i>Note: Do not start Tomcat yet!</i><br /><h4>Step 7 - Start Terracotta</h4>During the integration process you will want to have the Terracotta Developer Console running at all times. It will help you diagnose problems, and provide detailed statistics about Hibernate usage. Start it now:<blockquote style="padding-top: 10px; padding-right: 10px; padding-bottom: 10px; padding-left: 10px; border-top-width: 1px; border-right-width: 1px; border-bottom-width: 1px; border-left-width: 1px; border-top-color: rgb(221, 221, 221); border-right-color: rgb(221, 221, 221); border-bottom-color: rgb(221, 221, 221); border-left-color: rgb(221, 221, 221); border-top-style: dashed; border-right-style: dashed; border-bottom-style: dashed; border-left-style: dashed; ">$ $TC_HOME/bin/dev-console.sh</blockquote><br /><br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhtMuwx1cwE2CfzVueig2nSmMd13iJBdLBuKqQt4oEmihPkW5i8bym2h6h7PXXARLkxNd24rZl8xSWyr8-jlYaK6B_911RdsnM1jQNn1Oj0MYTXvFlArh-o9LNduv1by8fiB4xO/s1600-h/Screen+shot+2010-01-24+at+11.30.53+AM.png"><img style="cursor:pointer; cursor:hand;width: 400px; height: 289px;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhtMuwx1cwE2CfzVueig2nSmMd13iJBdLBuKqQt4oEmihPkW5i8bym2h6h7PXXARLkxNd24rZl8xSWyr8-jlYaK6B_911RdsnM1jQNn1Oj0MYTXvFlArh-o9LNduv1by8fiB4xO/s400/Screen+shot+2010-01-24+at+11.30.53+AM.png" border="0" alt=""id="BLOGGER_PHOTO_ID_5430425790139862226" /></a><br /><br />If the "Connect Automatically" box is not checked, check it now. <br /><br />Now you will need to start the Terracotta server:<blockquote style="padding-top: 10px; padding-right: 10px; padding-bottom: 10px; padding-left: 10px; border-top-width: 1px; border-right-width: 1px; border-bottom-width: 1px; border-left-width: 1px; border-top-color: rgb(221, 221, 221); border-right-color: rgb(221, 221, 221); border-bottom-color: rgb(221, 221, 221); border-left-color: rgb(221, 221, 221); border-top-style: dashed; border-right-style: dashed; border-bottom-style: dashed; border-left-style: dashed; "><span class="Apple-style-span" style="font-family:'courier new';">$ $TC_HOME/bin/start-tc-server.sh</span></blockquote>Your Developer Console should connect to the server and you should now see:<br /><br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj1HB4QK3McFf1ZuEUG6ZT4IoVMvzGC5RW-Cq_nZaAOwdkU8FKpi-p2APawmSXW47-2IHfM0OyMIl7Ef4QVz2jwBOQf2YvezLnASXdQyrzDrDXNnhEliOI5KcdQ-Hev5aAVk-rh/s1600-h/Screen+shot+2010-01-24+at+11.31.31+AM.png"><img style="cursor:pointer; cursor:hand;width: 400px; height: 289px;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj1HB4QK3McFf1ZuEUG6ZT4IoVMvzGC5RW-Cq_nZaAOwdkU8FKpi-p2APawmSXW47-2IHfM0OyMIl7Ef4QVz2jwBOQf2YvezLnASXdQyrzDrDXNnhEliOI5KcdQ-Hev5aAVk-rh/s400/Screen+shot+2010-01-24+at+11.31.31+AM.png" border="0" alt=""id="BLOGGER_PHOTO_ID_5430426127100251874" /></a><br /><br /><h4>Step 8 - Start Tomcat and PetClinic with Caching</h4><blockquote style="padding-top: 10px; padding-right: 10px; padding-bottom: 10px; padding-left: 10px; border-top-width: 1px; border-right-width: 1px; border-bottom-width: 1px; border-left-width: 1px; border-top-color: rgb(221, 221, 221); border-right-color: rgb(221, 221, 221); border-bottom-color: rgb(221, 221, 221); border-left-color: rgb(221, 221, 221); border-top-style: dashed; border-right-style: dashed; border-bottom-style: dashed; border-left-style: dashed; ">$ $TOMCAT_HOME/bin/catalina.sh start</blockquote><br /><br />Now access the PetClinic app again at <a href="http://localhost:8080/petclinic" title="http://localhost:8080/petclinic">http://localhost:8080/petclinic</a>. Take a look at the Developer Console. It should indicate one client has connected - this is the PetClinic app. Your Developer Console should look like this:<br /><br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEigHStEPcV27IN0bZ6J-EUM7i02eHM_Q6UXsRKatrQXMxjBAFwO_iFQ6WNK8zWCuTIocdPlD0ovwXAT0_0cY9CkhSqDVTNOkzCw-6fr4wWIk6mmm9lNQVo52RPkYm7FR3_euOOQ/s1600-h/Screen+shot+2010-01-24+at+11.37.10+AM.png"><img style="cursor:pointer; cursor:hand;width: 400px; height: 289px;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEigHStEPcV27IN0bZ6J-EUM7i02eHM_Q6UXsRKatrQXMxjBAFwO_iFQ6WNK8zWCuTIocdPlD0ovwXAT0_0cY9CkhSqDVTNOkzCw-6fr4wWIk6mmm9lNQVo52RPkYm7FR3_euOOQ/s400/Screen+shot+2010-01-24+at+11.37.10+AM.png" border="0" alt=""id="BLOGGER_PHOTO_ID_5430426747372502274" /></a><br /><br />If it does not, please review Steps 1-8. <br /><br />It's now time to review our application. Click the "Hibernate" entry in the Developer Console. You should now see the PetClinic entities listed. As you access entities in the application, the statistics in the Developer Console will reflect that access. Select "Refresh" to see an up to date view of the statistics.<br /><br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgPxriTHuDGNHxyzdisnT9fFXzWGgkXVQCeWd71ubL8fdf7kRtngfya0H1rOK9c3sFeUWiy9141nfRtNJxkVmrZ1Uv23P7aye3GEBTpx16U6d9hUT1_re4nPF-cvo_q8Y25zJ-R/s1600-h/Screen+shot+2010-01-24+at+11.42.27+AM.png"><img style="cursor:pointer; cursor:hand;width: 400px; height: 186px;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgPxriTHuDGNHxyzdisnT9fFXzWGgkXVQCeWd71ubL8fdf7kRtngfya0H1rOK9c3sFeUWiy9141nfRtNJxkVmrZ1Uv23P7aye3GEBTpx16U6d9hUT1_re4nPF-cvo_q8Y25zJ-R/s400/Screen+shot+2010-01-24+at+11.42.27+AM.png" border="0" alt=""id="BLOGGER_PHOTO_ID_5430427008778338242" /></a><br /><h4>Step 9 - Analyze Cache Performance</h4>Now we can use the Developer Console to monitor the cache performance. <br /><br />Select the "Second-Level Cache" button from the upper right hand corner. To monitor performance in real-time you can use either the Overview tab or the Statistics tab. <br /><br />In the Spring PetClinic app, select "Find owner" from the main menu. You should see the following screen:<br /><br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh5kRLKKiSp2BVhXZSvuj-P0hJHXkyID-XtnAjmdd3RE4OkoDFnLhP5qnJ6-qqOsGR7_0YP4t-IqREXovQNDXMxccs__pvXJhRa2gmNF86zWx-3gcUvH8DPMZlqHbpFv3YHqwTm/s1600-h/Screen+shot+2010-01-24+at+11.57.12+AM.png"><img style="cursor:pointer; cursor:hand;width: 400px; height: 214px;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh5kRLKKiSp2BVhXZSvuj-P0hJHXkyID-XtnAjmdd3RE4OkoDFnLhP5qnJ6-qqOsGR7_0YP4t-IqREXovQNDXMxccs__pvXJhRa2gmNF86zWx-3gcUvH8DPMZlqHbpFv3YHqwTm/s400/Screen+shot+2010-01-24+at+11.57.12+AM.png" border="0" alt=""id="BLOGGER_PHOTO_ID_5430427383934820482" /></a><br /><br />Now find all owners by pressing the "FIND OWNERS" button. Without any parameters in the query box, this will search for and display all owners. The results should look like this:<br /><br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiZ9IevoYo51hW51ze6GNyumuc79f2o4vTcEGRS7l2RupSgMcKj5bcXCTU8-WaFG213UAUPyWOYLwpwwZ2sXmGlBOupWijwxwKlHVu150lI2mXf09KXfC50HdktCb3tU8qf9h9A/s1600-h/Screen+shot+2010-01-24+at+11.58.21+AM.png"><img style="cursor:pointer; cursor:hand;width: 400px; height: 342px;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiZ9IevoYo51hW51ze6GNyumuc79f2o4vTcEGRS7l2RupSgMcKj5bcXCTU8-WaFG213UAUPyWOYLwpwwZ2sXmGlBOupWijwxwKlHVu150lI2mXf09KXfC50HdktCb3tU8qf9h9A/s400/Screen+shot+2010-01-24+at+11.58.21+AM.png" border="0" alt=""id="BLOGGER_PHOTO_ID_5430427519682659074" /></a><br /><br />If you are using the Overview tab, you will see the cache behavior in real-time. Refresh the find owners page (re-send the form if your browser asks) and then quickly switch to the Developer Console. You should see something like the following:<br /><br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhM-YUcOdIQqX1srC4m8jC4kteOWCMEDL1Gts_CRdTqMrO08W8HGlyj1g01Qgn7yEqLwvZMdc8RQ54Yfaa-TJ1-859mcmp278KAkWdpKDeZLJSnKDai8jPhE_-O4JKrxhvlC6-B/s1600-h/Screen+shot+2010-01-24+at+11.59.52+AM.png"><img style="cursor:pointer; cursor:hand;width: 400px; height: 289px;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhM-YUcOdIQqX1srC4m8jC4kteOWCMEDL1Gts_CRdTqMrO08W8HGlyj1g01Qgn7yEqLwvZMdc8RQ54Yfaa-TJ1-859mcmp278KAkWdpKDeZLJSnKDai8jPhE_-O4JKrxhvlC6-B/s400/Screen+shot+2010-01-24+at+11.59.52+AM.png" border="0" alt=""id="BLOGGER_PHOTO_ID_5430427683712149426" /></a><br /><br />Try switching tabs to the Statistics tab and do the same thing. Notice that in the statistics tab, you get a history of the recent activity. After finding the owners again your screen should like the following:<br /><br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiozNOqqbSf-w7oz4SxaLokbHqeAPBiwuR3CazZsKHY56sd5KvZi3hMPh4Ewg-L2WScnAnCVDQ2R0dg-WhOtYanUg1r_kmLo-k8ry-cT67EQzCZ1tHDi5kkH9sAEfK9uRGUe2sr/s1600-h/Screen+shot+2010-01-25+at+8.25.53+AM.jpg"><img style="cursor:pointer; cursor:hand;width: 400px; height: 289px;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiozNOqqbSf-w7oz4SxaLokbHqeAPBiwuR3CazZsKHY56sd5KvZi3hMPh4Ewg-L2WScnAnCVDQ2R0dg-WhOtYanUg1r_kmLo-k8ry-cT67EQzCZ1tHDi5kkH9sAEfK9uRGUe2sr/s400/Screen+shot+2010-01-25+at+8.25.53+AM.jpg" border="0" alt=""id="BLOGGER_PHOTO_ID_5430720922422256338" /></a><br /><br />Notice that there is a graph labeled "DB SQL Execution Rate". This graph shows exactly how many SQL statements are being sent to the database from Terracotta. This is a feature unique to Terracotta, because Terracotta adds special instrumentation into Hibernate that allows it to detect the DB SQL statements being sent to the DB. Let's use this feature to eliminate all database reads.<br /><h4>Step 10 - Eliminate all database activity</h4>By using the Developer Console we can see that we've eliminated almost all of the activity that we can reasonably expect. Of course we cannot eliminate the initial cache load, as the data must get into the cache somehow. So what are the little blips of database activity occurring after the initial load?<br /><br />The application activity that is causing the database activity is that of repeatedly listing all of the owners. Is it possible that this is causing our database activity - even though we've already cached all of the owners? <br /><br />Indeed, this is the case. To generate the list of owners, Hibernate must issue a query to the database. Once the result set is generated (a list of entity ids) all of the reads can subsequently be satisfied from cache. We can confirm that this is the case using the Developer Console - If you suspect that it can show you statistics on queries then you are starting to understand how the Developer Console works and what it can do for you!<br /><br />To see Hibernate query statistics, select the "Hibernate" button in the upper-right corner. Then select the "Queries" tab. This will show you the queries that are being performed by Hibernate. If we do so now, sure enough, just as we expected, we can see an entry for our Owner query:<br /><br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhfp455veYibCnhGjiSicPaAN0sHOIfBZmDjDdsoV7SQY297G8MCZhUMJbHv0GvBfVllarHuRr48Ikrzbclus4b9VG_qla_NiBFDwNbVPCw_3WH8NASL4L9LMQWDWcrREjaQ35j/s1600-h/Screen+shot+2010-01-24+at+12.13.50+PM.png"><img style="cursor:pointer; cursor:hand;width: 400px; height: 232px;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhfp455veYibCnhGjiSicPaAN0sHOIfBZmDjDdsoV7SQY297G8MCZhUMJbHv0GvBfVllarHuRr48Ikrzbclus4b9VG_qla_NiBFDwNbVPCw_3WH8NASL4L9LMQWDWcrREjaQ35j/s400/Screen+shot+2010-01-24+at+12.13.50+PM.png" border="0" alt=""id="BLOGGER_PHOTO_ID_5430427870025758914" /></a><br /><br />If we refresh our owner list again, and press "Refresh" on the query statistics, we should see the number in the Executions column increase by one. In my case, you see the number 5 in the previous screenshot. After reloading the owner query and refreshing the statistics page, I see the number 6.<br /><h4>Step 11 - Enable Query Caching</h4>Is there a way to eliminate these database queries? Yes, there is, it is called the Query Cache. To learn more about the query cache I highly recommend you read these resources:<br /><ul><li><a href="http://docs.jboss.org/hibernate/core/3.3/reference/en/html/performance.html" title="Hibernate Chapter 19 Performance Tuning">Hibernate Chapter 19 Performance Tuning</a></li><li >RJ Lorimer's excellent article <a href="http://www.javalobby.org/java/forums/t48846.html" title=""Hibernate: Truly Understanding the Second-Level and Query Caches"">"Hibernate: Truly Understanding the Second-Level and Query Caches"</a> on JavaLobby</li><li>Alex Miller's outstanding blog on understanding when to use the Query Cache <a href="http://tech.puredanger.com/2009/07/10/hibernate-query-cache/" title=""Hibernate query cache considered harmful?"">"Hibernate query cache considered harmful?"</a></li></ul><br />In short, the query cache can cache the results of a query, meaning that Hibernate doesn't have to go back to the database for every "list owners" request we make. Does it make sense to turn this caching on? Not always, as Alex points out (make sure you read his article). <br /><br />Because our goal today is to eliminate all database reads, we are going to turn on query caching. Whether you should do so or not for your application will depend on many factors, so make sure you fully understand the role of the query cache and how to use it.<br /><br />To enable query caching, we have to do two things:<br /><ul><li>Enable query caching in the hibernate config file</li><li>Enable caching for each query</li></ul><br />This isn't very different than how we enabled caching for entities, with one exception. To enable caching for each query, unfortunately we have to modify the code where the query is created.<br />So, to enable query caching in the hibernate config file, edit the applicationContext-hibernate.xml file again and add the hibernate.cache.use_query_cache prop:<br /><pre name="code" class="xml"><!-- Hibernate SessionFactory --><br /><bean id="sessionFactory" class="org.springframework.orm.hibernate3.LocalSessionFactoryBean" ref="dataSource" mappingresources="petclinic.hbm.xml"><br /> <property name="hibernateProperties"><br /> <props><br /> <prop key="hibernate.dialect">${hibernate.dialect}</prop><br /> <prop key="hibernate.show_sql">${hibernate.show_sql}</prop><br /> <prop key="hibernate.generate_statistics">${hibernate.generate_statistics}</prop><br /> <prop key="hibernate.cache.use_second_level_cache">true</prop><br /> <prop key="hibernate.cache.provider_class">org.terracotta.hibernate.TerracottaHibernateCacheProvider</prop><br /> <prop key="hibernate.cache.use_query_cache">true</prop><br /> </props><br /> </property><br /> <property name="eventListeners"><br /> <map><br /> <entry key="merge"><br /> <bean class="org.springframework.orm.hibernate3.support.IdTransferringMergeEventListener"><br /> </bean></entry><br /> </map><br /> </property><br /></bean></pre>Now, edit the source code. There is just one file to edit, located in src/org/springframework/samples/petclinic/hibernate/HibernateClinic.java. This file contains the definitions for all the queries. Edit it to add "setCacheable(true)" to each query like so:<br /> <br /><pre name="code" class="java">public class HibernateClinic implements Clinic {<br /> @Autowired<br /> private SessionFactory sessionFactory;<br /> @Transactional(readOnly = true)<br /> @SuppressWarnings("unchecked")<br /> public Collection >vet> getVets() {<br /> return sessionFactory.getCurrentSession().createQuery("from Vet vet order by vet.lastName, vet.firstName").setCacheable(true).list();<br /> }<br /><br /> @Transactional(readOnly = true)<br /> @SuppressWarnings("unchecked")<br /> public Collection <pettype> getPetTypes() {<br /> return sessionFactory.getCurrentSession().createQuery("from PetType type order by type.name").setCacheable(true).list();<br /> }<br /><br /> public Collection<owner> findOwners(String lastName) {<br /> return sessionFactory.getCurrentSession().createQuery("from Owner owner where owner.lastName like :lastName").setString("lastName", lastName + "%").setCacheable(true).list();<br />}</pre><br />Now re-build and re-deploy. <br /><br />Look in the Developer Console under the Second Level Cache/Statistics graph. There should be 0 DB executions (except the initial load).<br /><br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhvkk4Ee5mdIUaYHaPoL60bm9wRiytjcHb3w4rBObfW5rcet1ok-Dnl5PFMrruv5kegQHtVrnBft9ieYnL6P7in8XfHA_tNy5N7ukQobcI3v1mNHsLFSTaoyJmR0y7GDry1STtI/s1600-h/Screen+shot+2010-01-25+at+8.28.38+AM.jpg"><img style="cursor:pointer; cursor:hand;width: 400px; height: 289px;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhvkk4Ee5mdIUaYHaPoL60bm9wRiytjcHb3w4rBObfW5rcet1ok-Dnl5PFMrruv5kegQHtVrnBft9ieYnL6P7in8XfHA_tNy5N7ukQobcI3v1mNHsLFSTaoyJmR0y7GDry1STtI/s400/Screen+shot+2010-01-25+at+8.28.38+AM.jpg" border="0" alt=""id="BLOGGER_PHOTO_ID_5430719853161365618" /></a><br /><br /><h4>Conclusion</h4><br />I hope this tutorial was useful. Using caching in your application can improve performance and scale dramatically. If you'd like to review some performance numbers that Terracotta has published I recommend you visit <a href="http://www.terracotta.org/ehcache">Terracotta's EHCache site</a> and look in the right hand margin for a whitepaper that shows the results of performance testing the Spring PetClinic application.Taylorhttp://www.blogger.com/profile/07193759050963768511noreply@blogger.com12tag:blogger.com,1999:blog-35058937.post-9538683541107879542010-01-19T12:15:00.000-08:002010-01-20T14:17:22.616-08:00XTP Processing using a Distributed SEDA Grid built with CoherenceI just finished a talk at the <a href="http://blogs.oracle.com/csoto/2009/12/winter_2010_edition_of_the_new_york_coherence_sig.html">NYC Coherence SIG January 14, 2010</a>.<div><br /></div><div>The topic highlights the concepts <a href="http://www.griddynamics.com">Grid Dynamics used to build a high throughput scalable XTP</a> engine for processing telecommunications billing events. The framework employs a distributed SEDA architecture built on top of a Coherence In-Memory-Data-Grid back-end.</div><div><br /></div><div><a href="http://www.slideshare.net/ic6man/coherence-xtp-processing-using-seda">The slides are here. Enjoy!</a></div>Taylorhttp://www.blogger.com/profile/07193759050963768511noreply@blogger.com4tag:blogger.com,1999:blog-35058937.post-26862697359895380822010-01-03T16:47:00.000-08:002010-01-06T09:20:10.890-08:00Characterizing Enterprise Systems using the CAP theorem<p style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">In mid 2000, Eric A. Brewer, a former founder of Inktomi and chief scientist at Yahoo! and now currently a professor of Computer Science at U.C. Berkeley, presented a keynote speech at the ACM Symposium on the Principles of Distributed Computing. In his seminal speech, Brewer described a theorem based on research and observations he made called the <a href="http://www.julianbrowne.com/article/viewer/brewers-cap-theorem">CAP theorem</a>.</p><p style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><br />The CAP theorem is based on the observation that a distributed system is governed by three fundamental characteristics:</p><ol style="margin-top: 0px; margin-bottom: 0px; "><li style="margin-top: 0px; margin-bottom: 0px; ">Consistency</li><li style="margin-top: 0px; margin-bottom: 0px; ">Availability</li><li style="margin-top: 0px; margin-bottom: 0px; ">Partition tolerance</li></ol><p style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><br /></p><p style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">CAP is a is a useful tool in understanding the behavior of a distributed system. It states that given the three fundamental characteristics of a distributed computing system, you may have any two but never all three. It's usefulness in designing and building distributed systems cannot be overstated. So, how can we use the knowledge of this theorem to our advantage?</p><p style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><br /></p><p style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">As the designer of an enterprise scale system, CAP provides us with a framework to make decisions regarding which tradeoffs must be made in our own implementations. But CAP allows us not only to understand the systems we are building more precisely, but also provides a framework by which we can classify <i>all</i> systems. It is thus an invaluable tool when evaluating the systems that we rely on day in and day out in our enterprise systems.</p><br /><p style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">As an example, let's analyze the traditional <a id="cj-p" href="http://en.wikipedia.org/wiki/Relational_database_management_system" title="Relational Database Management System (RDBMS)" style="color: rgb(85, 26, 139); ">Relational Database Management System (RDBMS)</a>. The RDBMS, arguably one of the most successful enterprise technologies in history, has been around in its current form for nearly 40 years! The primary reason for the staying power of the RDBMS lies with its ability to provide consistency. A consistent system is most easily understood and reasoned about, and therefore most readily adopted (thus explaining the popularity of the RDBMS). But what of the other properties? An RDBMS provides availability, but only when there is connectivity between the client accessing the RDBMS and the RDBMS itself. Thus it can be said that the RDBMS does not provide partition tolerance - if a partition arises between the client and the RDBMS, the system will not be able to function properly. In summary, we can thus characterize the RDBMS as a <b>CA</b> system due to the fact that it provides <b>C</b>onsistency and <b>A</b>vailability but not <b>P</b>artition tolerance.</p><p style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><br /></p><p style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">As useful as this mechanism is, we can go one step further. Given that a system will always lack one of C, A, or P, it is common that mature systems have evolved a means of partially recovering the lost CAP characteristic. In the case of our RDBMS example, there are several well-known approaches that can be employed to compensate for the lack of <b>P</b>artition tolerance. One of these approaches is commonly referred to as master/slave replication. In this scheme, database writes are directed to a specially designated system, or master. Data from the master is then replicated to one or more additional, or slave, systems. If the master is offline then reads may be failed over to any one of the surviving read replica slaves. </p><br /><p style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">Thus, in addition to characterizing systems by their CAP traits, we can further characterize them by identifying the recovery mechanism(s) they provide for the lacking CAP trait. In the remainder of this article I classify a number of popular systems in use today in enterprise, and non-enterprise, distributed systems. These systems are:</p><p style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "></p><ul style="margin-top: 0px; margin-bottom: 0px; "><li style="margin-top: 0px; margin-bottom: 0px; ">RDBMS</li><li style="margin-top: 0px; margin-bottom: 0px; ">Amazon Dynamo</li><li style="margin-top: 0px; margin-bottom: 0px; ">Terracotta</li><li style="margin-top: 0px; margin-bottom: 0px; ">Oracle Coherence</li><li style="margin-top: 0px; margin-bottom: 0px; ">GigaSpaces</li><li style="margin-top: 0px; margin-bottom: 0px; ">Cassandra</li><li style="margin-top: 0px; margin-bottom: 0px; ">CouchDB</li><li style="margin-top: 0px; margin-bottom: 0px; ">Voldemort</li><li style="margin-top: 0px; margin-bottom: 0px; ">Google BigTable</li></ul><div style="margin-top: 0px; margin-bottom: 0px; "><br /></div><p style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><b>RDBMS</b></p><p style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">CAP: CA</p><p style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "></p><p style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">Recovery Mechanisms: Master/Slave replication, Sharding</p><div style="margin-top: 0px; margin-bottom: 0px; "><br /></div><p style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "></p><p style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">RDBMS systems are fundamentally about providing availability and consistency of data. The gold standard of RDMBS updates, referred to as <a id="r6on" href="http://en.wikipedia.org/wiki/ACID" title="ACID" style="color: rgb(85, 26, 139); ">ACID</a>, governs the way in which consistent updates are recorded and persisted.</p><p style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><br /></p><p style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">Various means of improving RDBMS performance are available in commercial systems. Due to the maturity of the RDBMS, these mechanisms are well understood. For example, the consistency conflicting reads and writes during the course of a transaction is referred to as <a id="p57h" href="http://en.wikipedia.org/wiki/Isolation_(database_systems)" title="isolation levels" style="color: rgb(85, 26, 139); ">isolation levels</a>. The commonly accepted set of isolation levels, in decreasing order of consistency (and increasing order of performance), are:</p><p style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "></p><ul style="margin-top: 0px; margin-bottom: 0px; "><li style="margin-top: 0px; margin-bottom: 0px; ">SERIALIZABLE</li><li style="margin-top: 0px; margin-bottom: 0px; ">REPEATABLE READ</li><li style="margin-top: 0px; margin-bottom: 0px; ">READ COMMITTED</li><li style="margin-top: 0px; margin-bottom: 0px; ">READ UNCOMMITTED</li></ul><p style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "></p><p style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><br /></p><p style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">Recovery mechanisms:</p><ul style="margin-top: 0px; margin-bottom: 0px; "><li style="margin-top: 0px; margin-bottom: 0px; ">Master/Slave replication: A single master accepts writes, data is replicated to slaves. Data read from slaves may be slightly out of date, trading off some amount of <b>C</b>onsistency to provide <b>P</b>artition tolerance.</li><li style="margin-top: 0px; margin-bottom: 0px; "><a id="zdcq" href="http://en.wikipedia.org/wiki/Shard_(database_architecture)" title="Sharding" style="color: rgb(85, 26, 139); ">Sharding</a>: While not strictly limited to database systems, sharding is commonly used in conjunction with a database system. Sharding refers to the practice of separating the entire application into vertical slices which are 100% independent of one another. Once completed, sharding isolates failures of any one system into "swimlanes" and is one example of <a id="hp7:" href="http://akfpartners.com/techblog/2008/05/30/fault-isolative-architectures-or-%E2%80%9Cswimlaning%E2%80%9D/" title=""fault isolative architectures"">"fault isolative architectures"</a>, thus limiting the impact of any single failure or related sets of failure to only one portion of an application. Sharding provides some measure of resistance to Partition tolerance by assuming that failures occur on a small enough scale to be isolated to a single shard, leaving the remaining shards operational.</li></ul><div style="margin-top: 0px; margin-bottom: 0px; "><br /></div><p style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><b>Amazon Dynamo</b></p><p style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">CAP: AP</p><p style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">Recovery: Read-repair, application hooks</p><div style="margin-top: 0px; margin-bottom: 0px; "><br /></div><p style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "></p><p style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><a id="z0p4" href="http://www.allthingsdistributed.com/2007/10/amazons_dynamo.html" title="Amazon's Dynamo," style="color: rgb(85, 26, 139); ">Amazon's Dynamo</a> is a private system designed and used solely by Amazon. Dynamo was intentionally designed to provide <b>A</b>vailability and <b>P</b>artitioning tolerance, but not <b>C</b>onsistency. This appearance of Amazon's Dynamo was very nearly as seminal as the introduction of the CAP theorem itself. Due to the dominance of the database, until Amazon introduced Dynamo to the world, it was very nearly a mainstay that enterprise systems must provide <b>C</b>onsistency and therefore the tradeoffs available lie in the remaining two CAP characteristics of <b>A</b>vailability or <b>P</b>artition tolerance. </p><p style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><br /></p><p style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">Examining the requirements for Amazon's Dynamo, it's clear why the designers chose to buck the trend: Amazon's business model depends heavily on availability. Even <a id="jejl" href="http://www.geek.com/articles/consumer/how-much-does-downtime-cost-20080710/" title="n the simplest of estimates" style="color: rgb(85, 26, 139); ">the simplest of estimates</a> pegs the losses Amazon could suffer from an outage at a minimum of $30,000 per <i>minute. </i>Given that Amazon's growth has nearly quadrupled since these estimates were made (in 2008), we can estimate that in 2010 Amazon may lose as much as <i>$100,000</i> per minute. Put simply, availability matters <i>a lot</i> at Amazon. Furthermore, the fallacies of distributed computing <a id="w-bx" href="http://en.wikipedia.org/wiki/Fallacies_of_Distributed_Computing" title="already know" style="color: rgb(85, 26, 139); ">already know</a> tells us that the network is unreliable, and so therefore we must expect partitions to occur on a regular and frequent basis. So it's a simple matter then to see that the only remaining CAP characteristic left to sacrifice is <b>C</b>onsistency.</p><p style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><br /></p><p style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "></p><p style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">Dynamo provides an eventual consistency model, where all nodes will <b>eventually</b> get all updates during their lifetime.</p><p style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">Given a system composed of N nodes, the eventual consistency model is tuned as follows:</p><ul style="margin-top: 0px; margin-bottom: 0px; "><li style="margin-top: 0px; margin-bottom: 0px; ">Setting the number of writes needed for a successful write operation (W).</li><li style="margin-top: 0px; margin-bottom: 0px; ">Setting the number of reads needed for a successful write operation (R).</li></ul><div style="margin-top: 0px; margin-bottom: 0px; "><br /></div>Setting W = N or R = N will give you a quorum-like system with strict consistency and no partition tolerance. Setting W <><div style="margin-top: 0px; margin-bottom: 0px; "><p style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><br /></p><p style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">Given that different nodes may have different versions of the same value (i.e., a value may have been written during a node downtime), Dynamo needs to:</p><ul style="margin-top: 0px; margin-bottom: 0px; "><li style="margin-top: 0px; margin-bottom: 0px; ">Track versions and resolve conflicts.</li><li style="margin-top: 0px; margin-bottom: 0px; ">Propagate new values.</li></ul>Versioning is implemented by using vector clocks: each value is associated to a list of (node, value) pairs updated every time a specific node writes that value; they can be used to determine causal ordering and branching. Conflict resolution is done during reads (read repair), eventually merging values with diverging vector clocks and writing back.</div><div style="margin-top: 0px; margin-bottom: 0px; "><br />New values are propagated by using hinted handoff and merkle trees.<br /><p style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "></p><p style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><br /></p><p style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><b>Terracotta</b></p><p style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">CAP: CA</p><p style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "></p><p style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">Recovery: Quorum vote, majority partition survival</p><p style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><br /></p><p style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><a id="dw6s" href="http://www.terracotta.org/" title="Terracotta" style="color: rgb(85, 26, 139); ">Terracotta</a> is a Java-based distributed computing platform that provides high level features such as Caching via EHCache and highly available scheduling via Quartz. Additional support for Hibernate second level caching allows architects to easily adopt Terracotta in a standard JEE architecture that relies on Spring, Hibernate and an RDBMS.</p><p style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><br /></p><p style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">Terracotta's architecture is similar to that of a database. Clients connect to one or more servers arranged into a "Server Array" layer. Updates are always Consistent in a Terracotta cluster, and availability is guaranteed so long as no partitions exist in the topology.</p><p style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><br /></p><p style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">Recovery Mechanisms:</p><p style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "></p><ul style="margin-top: 0px; margin-bottom: 0px; "><li style="margin-top: 0px; margin-bottom: 0px; ">Quorum: Upon failure of a single server, a backup server may take over once it has received enough votes from cluster members to elect itself the new master. </li><li style="margin-top: 0px; margin-bottom: 0px; ">Majority partition survival: In the event of a catastrophic partition involving many members of a Terracotta cluster that divides the cluster into one or more non-communicative partitions, the partition with a majority of remaining nodes is allowed to continue after a pre-configured period of time elapses.</li></ul><div style="margin-top: 0px; margin-bottom: 0px; "><br /></div><p style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "></p><p style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><b>Oracle Coherence</b></p><p style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">CAP: CA</p><p style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">Recovery: Partitioning, Read-replicas</p><p style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><br /></p><p style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><a id="v:eq" href="http://www.oracle.com/technology/products/coherence/index.html" title="Oracle Coherence" style="color: rgb(85, 26, 139); ">Oracle Coherence</a> is an in-memory Java data-grid and caching framework. It's main architectural component is its ability to provide consistency (hence it's name). All data in Oracle Coherence has at most one home. Data may be replicated to a configurable number of additional members in the cluster. When a system fails, replica systems vote on who becomes the new home for data that was homed in the failed system.</p><p style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><br /></p><p style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">Coherence provides data-grid features that facilitate processing data using map-reduce like techniques (execute the work on the data, instead of moving data to the processing) and a host of distributed computing patterns are available in the <a id="g3c4" href="http://coherence.oracle.com/display/INCUBATOR/Home" title="incubator patterns">incubator patterns</a>.</p><p style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><br /></p><p style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "></p><p style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">Recovery Mechanism(s): </p><p style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "></p><ul style="margin-top: 0px; margin-bottom: 0px; "><li style="margin-top: 0px; margin-bottom: 0px; ">Data Partitioning: At a granular level, any one piece of data exhibits CA properties, that is to say that reads and writes of data in Coherence are always <b>C</b>onsistent. As long as no partitions exist, data is available, meaning that for a particular piece of data, Coherence is not <b>P</b>artition tolerant. However, similar to the database sharding mechanism data may be partitioned across the cluster nodes, meaning that a partition will only affect a sub-set of all data. </li><li style="margin-top: 0px; margin-bottom: 0px; ">Read-replication: Coherence caches may be configured in <a id="qqy8" href="http://wiki.tangosol.com/display/COH35UG/Types+of+Caches+in+Coherence" title="varying topologies" style="color: rgb(85, 26, 139); ">varying topologies</a>. When a Coherence cache is configured in read-replicated mode it exhibits CA. Data is consistent but writes block in the face of partitions.</li></ul><p style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "></p><p style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "></p><p style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><br /></p><p style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><b>GigaSpaces</b></p><p style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">CAP: CA or AP, depending on the replication scheme chosen</p><p style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">Recovery: Per-key data partitioning</p><p style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><br /></p><p style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><a id="rrfu" href="http://www.gigaspaces.com/" title="GigaSpaces" style="color: rgb(85, 26, 139); ">GigaSpaces</a> is a Java based application server that is fundamentally built around the notion of Space-based computing, an idea derived from <a id="amyl" href="http://en.wikipedia.org/wiki/Tuple_space" title="Tuple Spaces" style="color: rgb(85, 26, 139); ">Tuple Spaces</a> which was the core foundation of the Linda programming system.</p><p style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><br /></p><p style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">GigaSpaces provides high availability of data placed in the space by means of synchronous and an asynchronous replication scheme. </p><p style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><br /></p><p style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">In a synchronous replication mode, GigaSpaces provides <b>C</b>onsistency and <b>A</b>vailability. The system is consistent and available, but can not tolerate partitions. In an asynchronous replication mode, GigaSpaces provides <b>A</b>vailability and <b>P</b>artition tolerance. The system is available for reads and writes, but is only eventually consistent (after the asynchronous replication completes).</p><p style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><br /></p><p style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">Recovery Mechanism(s):</p><p style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "></p><ul style="margin-top: 0px; margin-bottom: 0px; "><li style="margin-top: 0px; margin-bottom: 0px; ">Per-key data partitioning - GigaSpaces supports a mode called Partitioned-Sync2Backup. This allows for data to be partitioned based on a key to lower the risk of shared fate and to provide a synchronous copy for recovery.</li></ul><p style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "></p><p style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><br /></p><p style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><b>Apache Cassandra</b></p><p style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">CAP: AP</p><p style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">Recovery: Partitioning, Read-repair</p><p style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><br /></p><p style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><a id="prd-" href="http://incubator.apache.org/cassandra" title="Apache Cassandra was developed Facebook">Apache Cassandra</a> was developed by Facebook, using the same principles as Amazon's Dynamo, thus it is no surprise that Cassandra's CAP traits are the same as Dynamo's. </p><p style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><br /></p><p style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">For read-recovery, Cassandra uses simple timestamps instead of the more difficult vector clocks implementation used by Amazon's Dynamo.</p><p style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><br /></p><p style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">Recovery Mechanism(s):</p><p style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "></p><ul style="margin-top: 0px; margin-bottom: 0px; "><li style="margin-top: 0px; margin-bottom: 0px; ">Partitioning</li><li style="margin-top: 0px; margin-bottom: 0px; ">Read-repair</li></ul><p style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "></p><p style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><br /></p><p style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><b>Apache CouchDB</b></p><p style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">CAP: AP</p><p style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "></p><p style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">Recovery: </p><p style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><br /></p><p style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><a id="vh2j" href="http://couchdb.apache.org/" title="Apache CouchDB" style="color: rgb(85, 26, 139); ">Apache CouchDB</a> is a document oriented database that is written in Erlang.</p><div style="margin-top: 0px; margin-bottom: 0px; "><br /></div><p style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "></p><p style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><b>Voldemort</b></p><p style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">Link: http://project-voldemort.com</p><p style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">CAP: AP</p><p style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">Recovery: Configurable read-repair</p><p style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><br /></p><p style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">Project Voldemort is an open-source distributed key value store developed by LinkedIn and released as open source in February of 2009. Voldemort exhibits similar characteristics as Amazon's Dynamo. It uses vector clocks for version detection and read-repair.</p><p style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><br /></p></div><div style="margin-top: 0px; margin-bottom: 0px; "><p style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">Recovery Mechanism(s):</p><ul style="margin-top: 0px; margin-bottom: 0px; "><li style="margin-top: 0px; margin-bottom: 0px; ">Read-repair with versioning using vector clocks.</li></ul><p style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "></p></div><div style="margin-top: 0px; margin-bottom: 0px; "><p style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><br /></p><p style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><b>Google BigTable</b></p><b><p style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><span style="font-weight: normal; ">CAP: CA</span></p><div style="margin-top: 0px; margin-bottom: 0px; "><span style="font-weight: normal; "><p style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">Recovery: </p><p style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><br /></p><p style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><a id="ttj:" href="http://labs.google.com/papers/bigtable.html" title="Google's BigTable" style="color: rgb(85, 26, 139); ">Google's BigTable</a> is, <a id="zuyb" href="http://en.wikipedia.org/wiki/BigTable" title="according to Wikipedia," style="color: rgb(85, 26, 139); ">according to Wikipedia,</a> "a sparse, distributed multi-dimensional sorted map", sharing characteristics of both row-oriented and column-oriented databases." It relies on Google File System (GFS) for data replication. </p><div style="margin-top: 0px; margin-bottom: 0px; "><br /></div><div style="margin-top: 0px; margin-bottom: 0px; "><br /></div><div style="margin-top: 0px; margin-bottom: 0px; ">This blog post is still a work in progress. There are many other systems that are worthwhile to evaluate, among them <a href="http://code.google.com/p/terrastore/">Terrastore</a>, Erlang based frameworks like <a href="http://ftp.sunet.se/pub/lang/erlang/doc/apps/mnesia/users_guide.html">Mnesia</a>, and message based systems such as <a href="http://www.scala-lang.org/node/242">Scala Actors</a>, and <a href="http://doc.akkasource.org/">Akka</a>, among others. If you would like to see something else, please mention it in the comments.</div><div style="margin-top: 0px; margin-bottom: 0px; "><br /></div><div style="margin-top: 0px; margin-bottom: 0px; ">Thanks also go to <a href="http://sbtourist.blogspot.com/">Sergio Bossa</a> for assistance in writing this blog post.</div></span></div></b></div>Taylorhttp://www.blogger.com/profile/07193759050963768511noreply@blogger.com28tag:blogger.com,1999:blog-35058937.post-16864196325046860072009-10-01T08:45:00.001-07:002009-10-02T06:58:07.336-07:00A simple load test in Terracotta...This is a response to the following blog in which the author wrote a micro-benchmark and got some pretty bad results using Terracotta <a href="http://zion-city.blogspot.com/2009/10/terracotta-as-distributed-dbms-bad-idea.html">http://zion-city.blogspot.com/2009/10/terracotta-as-distributed-dbms-bad-idea.html</a>.<br /><br />Since the commenting system on blogger doesn't allow code, I am posting the response on my blog with code attached for reference.<br /><br />So my approach was to try replicate the author's implementation, to see what kind of performance a straightforward micro-benchmark might achieve.<br /><br /><i>Reader beware - micro-benchmarks are never a good idea, and not usually indicative of real-world performance. In this case, based on real-world results I have seen, my results appear to be a lower bound for the kind of performance one should expect since the test isn't concurrent and is running on a single machine - hardly the kind of environment a real world clustered app would exist in)</i><br /><br />So, with that said, I wrote a simple load test against a ConcurrentHashMap, and put 100,000 objects into it.<br /><br /><b>My results show:</b><br />Avg TPS: ~3,000<br />Instantaneous TPS as high as: ~7,000<br /><br />Here's the code:<pre name="code" class="java">import java.util.Date;<br />import java.util.Map;<br />import java.util.concurrent.*;<br /><br />public class Main<br />{<br /> static Map<Integer, Foo> map = <br /> new ConcurrentHashMap<Integer, Foo>();<br /><br /> public static class Foo<br /> {<br /> public String name;<br /> public String name2;<br /> public String name3;<br /><br /> public Foo(String name)<br /> {<br /> this.name = name;<br /> this.name2 = name + " 2";<br /> this.name3 = name + " 3";<br /> }<br /> }<br /><br /> public static void main(String[] args)<br /> {<br /> long start = System.currentTimeMillis();<br /><br /> for (int i = 0; i < 100000; i++) {<br /> map.put(i, new Foo(new Date().toString()));<br /> }<br /> System.out.println("elapsed: " + (System.currentTimeMillis() - start));<br /> }<br />}<br /></pre><br /><br />And here's the tc-config.xml:<br /><pre name="code" class="xml"><?xml version="1.0" encoding="UTF-8"?><br /><tc:tc-config xmlns:tc="http://www.terracotta.org/config"<br /> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"<br /> xsi:schemaLocation="http://www.terracotta.org/schema/terracotta-5.xsd"><br /> <application><br /> <dso><br /> <instrumented-classes><br /> <include><br /> <class-expression>Main$Foo</class-expression><br /> </include><br /> </instrumented-classes><br /> <roots><br /> <root><br /> <field-name>Main.map</field-name><br /> </root><br /> </roots><br /> </dso><br /> </application><br /></tc:tc-config><br /></pre><br /><br />I took a screenshot of the dev console running during the test, to give you an idea of the instantaneous TPS achieved:<br /><br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiKJUGtRNzIgtBYZu6oFfmu3yopZVcQzZpCryUB3XYJ0OTguSQogatAp97pME-SYQxMawGy0zDv7j6qXoQ56JfVX-bfajcgEgkyBKYEwaVn2YhB8_zNM4QE2GjX6-shx8eNCN-w/s1600-h/Screen+shot+2009-10-01+at+8.52.26+AM.png"><img style="cursor:pointer; cursor:hand;width: 400px; height: 312px;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiKJUGtRNzIgtBYZu6oFfmu3yopZVcQzZpCryUB3XYJ0OTguSQogatAp97pME-SYQxMawGy0zDv7j6qXoQ56JfVX-bfajcgEgkyBKYEwaVn2YhB8_zNM4QE2GjX6-shx8eNCN-w/s400/Screen+shot+2009-10-01+at+8.52.26+AM.png" border="0" alt=""id="BLOGGER_PHOTO_ID_5387660358159234658" /></a>Taylorhttp://www.blogger.com/profile/07193759050963768511noreply@blogger.com4tag:blogger.com,1999:blog-35058937.post-80132637294275607732009-09-02T19:38:00.000-07:002009-09-03T00:00:54.454-07:00Great customer service in the cloudIt's interesting to see providers moving to and, by proxy, the rest of us relying on, the cloud.<br /><br />I just spent a few hours at <a href="http://www.vmworld2009.com/">VMWorld</a>, and judging by the size, sophistication, and variety of providers, vendors, products, and companies, virtualization technologies, and in particular, cloud computing, is here to stay.<br /><br />Today, Google apologized for it's GMail outage yesterday, with a completely forthright, mature, and encouraging response. <br /><br /><div style="margin-left: 10px; font-style: italic;"><br />Gmail's web interface had a widespread outage earlier today, lasting about 100 minutes. We know how many people rely on Gmail for personal and professional communications, and we take it very seriously when there's a problem with the service. Thus, right up front, I'd like to apologize to all of you — today's outage was a Big Deal, and we're treating it as such. <a href="http://gmailblog.blogspot.com/2009/09/more-on-todays-gmail-issue.html"><read the rest of Google's post...></a><br /></div><br /><br />And, in a twist of fate, just a few days ago I received an email from Netflix. Apparently they had some trouble with their network while I was trying to watch a tv show using my XBox 360. Not only did they figure this out, but they sent me an email that offered to discount my monthly fee by 3%. This is fantastic customer service! Here's the email in it's entirety:<br /><br /><div style="margin-left: 10px; font-style: italic;"><br /><b>We're sorry you may have had trouble watching instantly via your Xbox</b><br /><br /> Dear Taylor,<br /><br />Last night, you may have had trouble instantly watching movies or TV episodes via your Xbox due to technical issues.<br /><br />We are sorry for the inconvenience this may have caused. If you were unable to instantly watch a movie or TV episode last night via your Xbox, click on this account specific link in the next 7 days to apply your 3% credit on your next billing statement. Credit can only be applied once.<br /><br />Again, we apologize for any inconvenience, and thank you for your understanding. If you need further assistance, please call us at 1-866-923-0898.<br /><br />-The Netflix Team<br /></div><br /><br />Failures do happen. And today's scaled-out architectures are designed to be resilient to these failures. But the fact is that even though these designs exist, and are generally very resilient against failures, giving these services availability times numbered in the <a href="http://en.wikipedia.org/wiki/High_availability">9's</a>, mistakes in design, implementation, or execution still do happen.<br /><br />I say, give these guys massive cred for owning up to their mistakes, and dealing with the consumer in an open and honest way. That's the way to build solid relationships, and I for one will not look twice when Yahoo! or Blockbuster sends me that next request to join up on their service. These guys have it figured out, and have sold me as lifelong* customer, even if they're not perfect.<br /><br />If only legacy infrastructure (power, cable (grrr Comcast), telephone (AT&T - I'm looking at you)) and the like could understand the value in this approach.<br /><br /><div style="font-size: 9pt">* lifelong in tech years, which is only about 5 years or so ;-)</div>Taylorhttp://www.blogger.com/profile/07193759050963768511noreply@blogger.com1tag:blogger.com,1999:blog-35058937.post-12332493411731501382009-06-17T01:37:00.001-07:002009-06-17T01:51:08.177-07:00How To Optimize Performance (or how to do Performance Testing right)Optimizing performance requires you to performance test.<br /><br />I'm just going to say it - performance testing is hard. Really hard.<br /><br />Ask anyone that's done it before, and they will agree. If you haven't done it before, well, yeah, sorry. It's not easy. But you've got to do it anyway - because the most important thing you will do as a software engineer is performance test. It's a bit like when your Dad told you "when you grow up to be my age <insert age old wisdom here>" and you didn't believe him? <br /><br />And now you're old enough that you realize, hey, the guy might have had a point?<br /><br />Yeah, trust me. Performance testing is both the hardest, and most important, thing you will ever do in your software engineering career. Get it right - you'll be a rockstar. Don't do it - well, I promise you, you'll always be griping about why the amazing software you write is never actually used for "production" apps.<br /><br />So here you go, simple steps to performance engineering:<br /><br />1) Set goals - what are you trying to accomplish<br />2) Measure a baseline<br />3) Identify a bottleneck<br />4) Fix said bottleneck<br />5) Repeat until you meet your performance goals<br /><br />Did I miss anything? Ahhh...yes. TAKE NOTES.<br /><br />Let's try again:<br /><br /><br />1) Set goals - what are you trying to accomplish<br />1a) Take notes<br />2) Measure a baseline<br />2a) Take notes<br />3) Identify a bottleneck<br />3a) Take notes<br />4) Fix said bottleneck<br />4a) Take notes<br />5) Repeat until you meet your performance goals<br /><br />Step 6) -- Report to your boss how much better your application is. But because of Step 1, you'll be able to tell him/her why it matters, right? :)Taylorhttp://www.blogger.com/profile/07193759050963768511noreply@blogger.com10tag:blogger.com,1999:blog-35058937.post-51219351466498603022009-04-24T14:31:00.000-07:002009-04-24T15:04:37.747-07:00Terracotta and Spring - Powering High Throughput JEE ApplicationsRecently, Terracotta did a webinar with Spring founder <a href="http://www.springsource.com/people/csampaleanu">Colin Sampaleanu</a>. <br /><br />The webinar starts out by covering the benefits that a Spring+Hibernate+Terracotta application can deliver for your Java JEE application. The latter half is dedicated to running through a reference application that provides a solid starting point as you explore all that Terracotta+Spring can provide.<br /><br /><span style="font-weight:bold;">Examinator</span><br /><br />The application demonstrated in the webinar is called "Examinator", and was jointly developed by SpringSource and Terracotta. Briefly, <br /><br /><div style="margin: 0ex 4ex; font-style: italic; font-weight: bold">Examinator is a full stack reference application which demonstrates with code how to build a higly scalable, highly available application using Spring, Hibernate and Terracotta</div><br /><br />Highlights include:<ul><li><span style="font-weight:bold;">Frameworks:</span> Spring MVC, Spring Security, Spring Web Flow, Hibernate</li><br /><li><span style="font-weight:bold;">Scale:</span> 16 application servers, 20,000 concurrent users</li><br /><li><span style="font-weight:bold;">Latency:</span> Max of 5 ms response time</li><br /></ul><span style="font-weight:bold;">Find out more</span><br /><br />For a full-length recording of the Webinar available for free visit <a href="https://terracotta.webex.com/terracotta/lsr.php?AT=pb&SP=EC&rID=1261917&rKey=35185C4070100891">http://terracottech.webex.com</a>.<br /><br />For a complete reference—everything you need to know including full documentation, how install and run Examinator— visit <a href="http://www.terracotta.org/web/display/orgsite/Web+App+Reference+Implementation">http://www.terracotta.org</a>. <br /><br />You can also access a live demo of Examinator at <a href="http://reference.terracotta.org/examinator/">reference.terracotta.org</a>.<br /><br /><span style="font-weight:bold;">SpringOne 2009</span><br /><br />Speaking of Spring, I'll be attending the Terracotta booth at <a href="http://europe.springone.com/europe-2009/">SpringOne Europe 2009 </a>this coming week (April 27-29, 2009) in Amsterdam. Stop by if you're attending.Taylorhttp://www.blogger.com/profile/07193759050963768511noreply@blogger.com1tag:blogger.com,1999:blog-35058937.post-24487410143195184532009-04-22T08:16:00.001-07:002009-04-22T13:56:42.679-07:00A simple tip for new Terracotta users - always run the Terracotta Developer ConsoleWith the release of Terracotta 3.0, I hope many of you have or are considering checking out Terracotta to see if it can help with scalability and availability of your Java application.<br /><br />Of course <a href="http://www.terracotta.org">www.terracotta.org</a> - in particular the <a href="http://www.terracotta.org/web/display/orgsite/Tutorials">tutorials</a> section with many simple recipes for exploring the many uses of Terracotta is a good place to get started.<br /><br />But before you do any of that, I'd like to point out a best practice for learning and working with Terracotta. So, here's my tip for whenever you are working with Terracotta:<br /><br /><center><b>TIP: ALWAYS RUN THE TERRACOTTA DEVELOPER CONSOLE</b></center><br />It's easy to do, so I recommend before you run any samples, try an recipes, or work with your application, make sure to have the Developer Console running at all times.<br /><h4>How to run the Terracotta Developer Console</h4><br />Running the Developer Console is easy. There are many ways depending on your context:<br /><ul><br /><li>From the Welcome Application: Click the "Developer Console" link</li><br /><li>From the command line: Run <code>$TC_HOME/bin/dev-console.sh|bat</code></li><br /><li>From Maven: Run <code>$ mvn tc:dev-console</code></li><br /><li>From Eclipse: Select <code>Show Developer Console</code> from the Terracotta menu.</li><br /></ul>Once you've got the Developer Console running, make sure to select the <code>Connect automatically</code> checkbox before connecting—this option will automatically connect the Developer Console to your cluster meaning you don't have to select "connect" every time you run a new cluster instance. This is very useful during experimentation (running sample demos and recipes) and integration testing. <br /><h4>Why should you run the Terracotta Developer Console?</h4>We designed it so that you have access to a large array of information right at your fingertips. In particular, let's look at the user interface which is new in Terracotta 3.0:<br /><br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjaVjW9mvAJmbjekpjIu1aIIVDslhpEBsX1Hri23PGND7cWlrXOOpWlmmTe7cS_yFyGM-fmQgG9-m8B21qZnENVhOk54YZN_APQjd3w19RtZP7VtgHBUzz2zoq_zGcSqRVr3I9t/s1600-h/Picture+1.png"><img style="margin:0 0 10px 10px;cursor:pointer; cursor:hand;width: 400px; height: 238px;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjaVjW9mvAJmbjekpjIu1aIIVDslhpEBsX1Hri23PGND7cWlrXOOpWlmmTe7cS_yFyGM-fmQgG9-m8B21qZnENVhOk54YZN_APQjd3w19RtZP7VtgHBUzz2zoq_zGcSqRVr3I9t/s400/Picture+1.png" border="0" alt=""id="BLOGGER_PHOTO_ID_5327536249245161938" /></a><br /><br />One thing that I hope jumps out at you immediately is the presence of a new array of "speedo" dials - somewhat like the array of instruments that greets you when you step into the driver's seat of an automobile.<br /><br />The resemblance isn't accidental. Those dials are there to give you up-to-the-second information about what's going on in the cluster - and to help pinpoint a problem - if there is one. Let's take a closer look:<br /><h4>Making use of the Speed Dials</h4><br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhnPfML8OPpwn0sS6FqI2pi5gGemIqUbB0Iy-61RulmETVxy8ENAR-UrD4sIzwMmtU27zqE8CGIxAYGF6A8UWX995VAhgCccQiJu6IAcWnGBrC5YzoyBcmGect03P2MEMl5ZhiJ/s1600-h/dials.png"><img style="cursor:pointer; cursor:hand;width: 400px; height: 72px;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhnPfML8OPpwn0sS6FqI2pi5gGemIqUbB0Iy-61RulmETVxy8ENAR-UrD4sIzwMmtU27zqE8CGIxAYGF6A8UWX995VAhgCccQiJu6IAcWnGBrC5YzoyBcmGect03P2MEMl5ZhiJ/s400/dials.png" border="0" alt=""id="BLOGGER_PHOTO_ID_5327537742582481314" /></a><br /><br />As you can see, the dials are arrayed from left to right, giving you vital statistics about the cluster. The dials are separated into two groups:<ul><br /><li>Write Transacions</li><br /><li>Impeding Factors</li><br /></ul>The <b>Write Transactions</b> dial contains a measure of the number of write transactions that are occurring in the system. Read transactions with Terracotta are exceedingly cheap (so cheap in fact that we don't measure them). Write transactions are a good measure of work being done in the cluster - so this measure is effectively a measure of how fast your application is running.<br /><br />The <b>Impeding Factors</b> set of dials shows you a set of seven dials that show you statistics about other types of activity in the system. The activities displayed include such statistics as <i>Object Creation Rate/s</i> — the amount of new objects being added to the clustered heap per second — and <i>Lock Recalls/s</i> — the amount of lock requests that are being transferred from one client node to another.<br /><h4>Making use of the Runtime Statistics</h4>Another very useful feature is the Runtime statistics panel. You can access this feature from the left menu tree by selecting the <code>Runtime Statistics</code> node.<br /><br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiG0SOAp7umB360u9xwnxVbkuQ3clmCB3sRiJQX606o9K7z5uqwHUYuoKEpic8f1KtFLSWmfLxaMPp7XaWpblWKzE5zHtmBRRwPIcM-8tL-bVDUsbeda-myVii8ZHLxvAxcBC2w/s1600-h/runtime+stats.png"><img style="cursor:pointer; cursor:hand;width: 400px; height: 219px;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiG0SOAp7umB360u9xwnxVbkuQ3clmCB3sRiJQX606o9K7z5uqwHUYuoKEpic8f1KtFLSWmfLxaMPp7XaWpblWKzE5zHtmBRRwPIcM-8tL-bVDUsbeda-myVii8ZHLxvAxcBC2w/s400/runtime+stats.png" border="0" alt=""id="BLOGGER_PHOTO_ID_5327542276603671506" /></a><br /><br />The runtime statistics give you access to a wealth of realtime data with historical views. Unlike the Speedo Dials, the runtime statistics are kept for a longer period of time and graphed for you so you can see a historical view of how your application has been behaving.<br /><h4>Putting it all together</h4>The Speed Dials give you instantaneous information - so they are visible all the time.<br /><br />Look at the <i>Write Transactions</i> to measure your speed, and monitor the <i>Impeding Factors</i> to make sure nothing is slowing you down. <br /><br /> If there's something worth looking at in more detail, switch to the runtime statistics for more detailed information. <br /><br />If there is a problem worth investigating, use the <i>Diagnostics</i> tools such as the <i>Lock Profiler</i> or <i>Cluster Wide Thread Dump</i> to debug further.<br /><br />In other words <b>ALWAYS RUN THE TERRACOTTA DEVELOPER CONSOLE!</b>Taylorhttp://www.blogger.com/profile/07193759050963768511noreply@blogger.com1tag:blogger.com,1999:blog-35058937.post-92149441659372371242008-12-14T10:45:00.000-08:002008-12-15T08:51:37.472-08:00Simple Java MessagingFollowing up on my recent post <a href="http://javathink.blogspot.com/2008/12/java-distributed-lock-manager.html">Java Distributed Lock Manager</a>, sometimes you just need a simple way to pass messages between Java processes. <br /><br />Messaging is a very useful pattern in Enterprise Integration, and there are many ways to do it. <a href="http://activemq.apache.org/camel/">Apache Camel</a> is a great tool when you need the flexibility and power to manage complex messaging patterns, including routing, filtering and the like.<br /><br />If you just want to do something simple, though, that can be a challenge. The most common solution, JMS, requires quite a bit of boilerplate code, and requires selecting and running a JMS provider, which means selecting a J2EE container, <a href="http://activemq.apache.org/">Apache ActiveMQ</a>, or others.<br /><br />So what if you just want a drop-dead <a href="http://www.terracotta.org">simple way of adding messaging to your application</a>? Terracotta gives you that. (And also integrates well with other solutions, like Apache Camel if you need more power later on). <br /><br />Simple messaging in Terracotta is built on the notion of clustering a <a href="http://java.sun.com/j2se/1.5.0/docs/api/java/util/concurrent/LinkedBlockingQueue.html">LinkedBlockingQueue</a>. Just as a LinkedBlockingQueue is used to pass messages between threads in a single JVM, it will be used in combination with Terracotta's JVM-level clustering to provide message passing between JVMs.<br /><br />To demonstrate, here is a simple example. <pre name="code" class="java">import java.io.*;<br />import java.util.concurrent.*;<br />import java.util.concurrent.locks.*;<br /><br />public class SimpleMessage<br />{<br /> private static ReentrantReadWriteLock lock = new ReentrantReadWriteLock();<br /> private static BlockingQueue<String> queue = new LinkedBlockingQueue<String>();<br /><br /> public static void receive() throws InterruptedException<br /> {<br /> System.out.println("Receiving messages...");<br /> while (true) {<br /> String msg = queue.take();<br /> System.out.println("msg >> " + msg);<br /> }<br /> }<br /><br /> public static void send() throws Exception<br /> {<br /> while (true) {<br /> System.out.print("Enter a message> "); System.out.flush();<br /> String msg = new BufferedReader(new InputStreamReader(System.in)).readLine();<br /> queue.put(msg);<br /> }<br /> }<br /><br /> public static void main(String[] args) throws Exception<br /> {<br /> // we use the presence of a lock to distinguish receiver from sender<br /> if (lock.writeLock().tryLock()) {<br /> receive();<br /> } else {<br /> send();<br /> }<br /> }<br />}</pre>The app consist of two modes - a receiver mode and a sender mode. Normally, you would have an application specific mechanism of choosing whether you wanted to send messages or receive messages. For this example, we use a simple lock (for more information on using a ReentrantReadWriteLock with Terracotta, read the <a href="http://www.terracotta.org/web/display/orgsite/Recipe?recipe=rrwl">ReentrantReadWriteLock recipe</a>). When free, the lock indicates no processes are receiving messages, so the process takes on the "receiver" mode. All subsequent processes take on the "sender" mode when the lock is held.<br /><br />So let's run it with Terracotta and see how it works. First, we need to "cluster" the app. We need the <code>lock</code> and <code>queue</code> objects to be the same cluster-wide, which in Terracotta is called a root. So our Terracotta configuration file looks like:<pre name="code" class="xml"><tc:tc-config xmlns:tc="http://www.terracotta.org/config"<br /> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"<br /> xsi:schemaLocation="http://www.terracotta.org/schema/terracotta-4.xsd"><br /><br /> <application><br /> <dso><br /> <roots><br /> <root><br /> <field-name>SimpleMessage.lock</field-name><br /> </root><br /> <root><br /> <field-name>SimpleMessage.queue</field-name><br /> </root><br /> </roots><br /> </dso><br /> </application><br /></tc:tc-config><br /></pre>Now, let's run two JVMs with Terracotta. First, we start a server instance:<pre>$ start-tc-server.sh<br />2008-12-14 10:26:18,246 INFO - Terracotta Server has started up as ACTIVE node on<br /> 0.0.0.0:9510 successfully, and is now ready for work.<br /></pre>Then, we start our JVMs. <br /><br /><b>JVM 1:</b><pre>$ dso-java.sh SimpleMessage<br />Receiving Messages...<br /></pre><b>JVM 2:</b><pre>$ dso-java.sh SimpleMessage<br />Enter a message> <br /></pre>Here, we enter a message, and see that it is printed in JVM 1:<br /><br /><b>JVM 2:</b><pre>$ dso-java.sh SimpleMessage<br />Enter a message> hello world<br /></pre><b>JVM 1:</b><pre>$ dso-java.sh SimpleMessage<br />Receiving Messages...<br />msg >> hello world<br /></pre><b>Further exploration</b><br /><br />Try starting another JVM and see that they can both send messages to JVM 1. Try killing the receiver JVM and send messages to it. Then start another JVM. Since the lock is no longer held (Terracotta automatically releases any locks held by a JVM that exits the cluster) the new JVM will take on the receiver mode. Any messages sent while there was no receiver will have been queued, and will be printed on the startup of this new node.<br /><br />And of course, you can see all the activity in the cluster. Try taking the receiver down again, send some messages using the sender nodes, then <a href="http://www.terracotta.org/web/display/docs/Admin+Console+Guide">run the admin console</a>. You'll be able to inspect the messages in the queue using the clustered heap browser.<br /><br />This is just a demonstration of course - so to keep it simple I used a <code>String</code> as the message - but you could use any class.<br /><br />For more fun with Terracotta, <a href="http://www.terracotta.org/web/display/orgsite/Tutorials">try the helpful "recipes" at Terracotta.org</a>.<br /><br />(Note, <a href="http://javathink.blogspot.com/2008/03/stupid-simple-jvm-coordination.html">I've blogged about simple coordination</a> in the past using Terracotta, which is similar)Taylorhttp://www.blogger.com/profile/07193759050963768511noreply@blogger.com17tag:blogger.com,1999:blog-35058937.post-70308635333252882272008-12-11T22:14:00.000-08:002009-03-17T02:20:05.713-07:00Java Distributed Lock ManagerSometimes you just need a simple way to coordinate activities across more than one java process. There's a lot of choices out there. The database, JMX, distributed caches, JMS, filesystems. It would be nice if there was a simple, easy way to get distributed locks in a J2SE, J2EE, Web, SOAP, or AJAX application? There is.<br /><br />Terracotta provides one of the <a href="http://www.terracotta.org">easiest ways to get a distributed lock manager in your Java application</a>. Terracotta plugs right in to normal Java threading constructs—<code>synchronized</code>, <code>wait/notify</code>, <code><a href="http://java.sun.com/j2se/1.5.0/docs/api/java/util/concurrent/locks/ReentrantReadWriteLock.html">java.concurrent.locks.ReentrantReadWriteLock</a></code>, and even <code><a href="http://java.sun.com/j2se/1.5.0/docs/api/java/util/concurrent/CyclicBarrier.html">java.concurrent.CyclicBarrier</a></code>, which means you basically already know how to use Terracotta as a lock manager.<br /><br />To demonstrate, let's work up a simple locking example and then drop Terracotta in. Our app will consist of acquiring a lock, "do some work" in a simple loop, and repeat. Here's the code (<code>LockExample.java</code>):<br /><pre name="code" class="java">public class LockExample<br />{<br /> private static Object lock = new Object();<br /><br /> public static void main(String[] args) throws Exception<br /> {<br /> while (true) {<br /> System.out.print("Waiting for the lock..."); System.out.flush();<br /> synchronized (lock) {<br /> System.out.print("I got the lock, doing work");<br /> for (int i = 0; i < 4; i++) {<br /> Thread.currentThread().sleep(1000);<br /> System.out.print("."); System.out.flush();<br /> }<br /> }<br /> System.out.println("done");<br /> }<br /> }<br />}</pre>Simple. If we run this on the command line, we get:<pre>$ javac LockExample.java<br />$ java LockExample<br />Waiting for the lock...I got the lock, doing work....done<br />Waiting for the lock...I got the lock, doing work....done<br /></pre>During the "work" part the "."'s are added 1 every second for four seconds. Fancy.<br /><br />Let's add Terracotta. We need a <code>tc-config.xml</code> file which tells Terracotta how to provide the appropriate clustering behavior to our application:<pre name="code" class="xml"><tc:tc-config xmlns:tc="http://www.terracotta.org/config"<br /> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"<br /> xsi:schemaLocation="http://www.terracotta.org/schema/terracotta-4.xsd"><br /><br /> <application><br /> <dso><br /> <locks><br /> <autolock><br /> <method-expression>void LockExample.main(..)</method-expression><br /> </autolock><br /> </locks><br /> <roots><br /> <root><br /> <field-name>LockExample.lock</field-name><br /> </root><br /> </roots><br /> </dso><br /> </application><br /></tc:tc-config><br /></pre>Now, let's run two JVMs with Terracotta. First, we start a server instance:<pre>$ start-tc-server.sh<br />2008-12-11 22:26:18,246 INFO - Terracotta Server has started up as ACTIVE node on<br /> 0.0.0.0:9510 successfully, and is now ready for work.<br /></pre>Then, we start our JVMs. <br /><br /><b>JVM 1:</b><pre>$ dso-java.sh LockExample<br />Waiting for the lock...I got the lock, doing work....done<br /></pre><b>JVM 2:</b><pre>$ dso-java.sh LockExample<br />Waiting for the lock...<br /></pre>It's a bit hard to demonstrate in a blog post, but the lock ping-pongs between the JVMs. That's it! <br /><br />For more fun with distributed lock coordination, try these helpful "recipes":<br /><ul><li><a href="http://www.terracotta.org/web/display/orgsite/Recipe?recipe=rrwl">ReentrantReadWriteLock</a></li><br /><li><a href="http://www.terracotta.org/web/display/orgsite/Recipe?recipe=cyclicbarrier">CyclicBarrier</a></li><br /></ul>Taylorhttp://www.blogger.com/profile/07193759050963768511noreply@blogger.com9tag:blogger.com,1999:blog-35058937.post-52155899604584184002008-11-22T20:38:00.001-08:002009-02-21T19:46:51.487-08:00Sales 101: The 5 sales archetypesThere's a lot they don't teach you in school. As an engineer, most of the things I rely on day to day they never even so much as mentioned when I was in college, things like bug tracking, revision control, heck even writing error messages.<br /><br />If you're an engineer, and in sales, it's even worse. I have spent the better half of my career in pre and post-sales as a "solutions architect" before moving on to product management. One thing that I have found to be useful is how to identify the person you are talking to on the other end of the phone.<br /><br />I am not talking about figuring out if you are talking to a developer, a manager, an architect or a CEO. I mean what kind of person is this - what problems do they have, what kind of ego do they have, but most importantly, are they actually going to spend money. Like it or not, these considerations make a major contrbution to chances of your success. <br /><br />So here's the 5 kinds of people you are most likely to meet, how to identify them, and what they mean to the bottom-line.<br /><br /><span style="font-weight:bold;">1. The beard-tugger</span><br /><br /><span style="font-style:italic;">Summary:</span> The beard-tugger thinks he is smarter than everyone else, and is committed to proving it. Thus he will spend the entire sales-pitch showing you just how smart he is. <br /><br /><span style="font-style:italic;">Tell tale signs you have a beard-tugger:</span> For every feature you talk about in your product, the beard-tugger will analyze it in 5 different ways and 10 different contexts. If he sees the slightest hole in your theory or implementation, he's bound to ask a question about it. <br /><br /><span style="font-style:italic;">Pros:</span> If you like to get into the guts of the product with someone, this is a great person to do it with. Remember though that if you are trying to sell this person, you need to let them "win" - e.g. never show them up to be lesser than the audience, or you risk losing them as a champion of your product.<br /><br /><span style="font-style:italic;">Cons:</span> There is no faster way to sink a pitch than to rat-hole on some minor detail of your product. The beard-tuggers middle name is rat-hole, so be careful.<br /><br /><span style="font-weight:bold;">2. The tire-kicker</span><br /><br /><span style="font-style:italic;">Summary:</span> The tire-kicker is out for a good time. They will take a sales call from anyone, and don't really have an agenda. They just want to see what you have. Often, this person thinks of themselves as "knowledgable about the market" so they will talk to you just to get a feel for your product to reinforce that feeling.<br /><br /><span style="font-style:italic;">Tell tale signs you have a tire-kicker:</span> if your product can be used in 10 different ways, he wants to know about every one of them. If you ask them about the specific problem they are looking to solve (and you should) the answer will be vague or none at all.<br /><br /><span style="font-style:italic;">Pros:</span> none really. This person may ultimately be a champion for you when they come around to solving a problem, so don't shun them. But you should get on with your life as soon as you can. Pitch your wares succinctly and move on.<br /><br /><span style="font-style:italic;">Cons:</span> big time suck. If you are not careful this person could come back time and time again, without a real problem to solve. That could be a big time sink for no real opportunity for business, in other words, a real waste of your time.<br /><br /><span style="font-weight:bold;">3. The science-experiment</span><br /><br /><span style="font-style:italic;">Summary:</span> Similar to the tire-kicker, but a little more involved, the science-experimenter is likely to have a problem in mind and wants to solve it, but there's generally no business behind it. This person is likely to be "exploring" technologies, but has picked yours as a likely candidate for a solution.<br /><br /><span style="font-style:italic;">Tell tale signs you have a science-experiment: </span> this person is allocating resources to a project to test out your product - e.g. run a POC (Proof of Concept). But if you press them, you will find there are no hard and fast requirements, so everything is either made up or guesswork. This person is likely to be enthusiastic both about their use case, and your product. They are probably even more enthusiastic about the opportunity for the mutual relationship to grow. <br /><br /><span style="font-style:italic;">Pros:</span> If you are successful, this could blossom into a sale, and the science-experimenter will probably tell you that at least 5 times in the span of 20 minutes. <br /><br /><span style="font-style:italic;">Cons: </span> A science-experiment goes nowhere 9 times out of 10. Stay away because there are no real requirements, and even if you succeed (which is unlikely, given the lack of a real requirement or business driver) there is unlikely to be a pot of gold at the end of this relationship, despite the claims of the science-experimenter to the contrary.<br /><br /><span style="font-weight:bold;">4. The delusional</span><br /><br /><span style="font-style:italic;">Summary: </span> This person is trying to use your product either for an outlandish use case, in an extreme way, or worse, in every possible way (e.g. they think you have the silver-bullet to every problem known to man).<br /><br /><span style="font-style:italic;">Tell tale signs you have a person with delusions of grandeur</span> on the line is that they will be obsessed with how successful their product <span style="font-style:italic;">will</span> be. That is to say, they have either no product, or a very small product at the moment, and this person is most likely obsessed with the massive growth they are just about to incur.<br /><br /><span style="font-style:italic;">Pros:</span> None. This is one of the most dangerous people to engage with. Get out fast.<br /><br /><span style="font-style:italic;">Cons:</span> Very likely to be enthusiastic with your product and want to POC it. This is the most dangerous of people, because they will be pandering to <span style="font-style:italic;">your</span> ego, which means you will be very easily swayed by them because they believe in you, and they love your product. They will spend a very long time in a POC with your product, because the requirements will likely be unrealistic bordering on ridiculous. If you bend to their whims, you will likely be adding features/fixing bugs that have no bearing on real business.<br /><br /><span style="font-weight:bold;">5. Your customer</span><br /><br /><span style="font-style:italic;">Summary: </span> This is the person you want to sell to. They have a business case, and likely some pain. When you discover what that pain is, you should have a product that solves that pain, for less money then they are currently spending. <br /><br /><span style="font-style:italic;">Tell tale signs you are talking to your ideal customer:</span> at the end of your initial call with this person, they should have asked you how much your product costs, and they should be wary that your product can actually solve their problem (afterall, nothing else has to date), but be slightly optimistic that maybe you can solve their problem, and be willing to try. They should have a use case that is within the bounds of what your product can solve, and they should be interested in what the next steps are after the call.<br /><br /><span style="font-style:italic;">Pros:</span> Sell them quickly - this is where you should be spending your time.<br /><br /><span style="font-style:italic;">Cons:</span> None - your only challenge is to identify this person. If you don't, you're losing money.<br /><br />Now, go out there and make some money! :)Taylorhttp://www.blogger.com/profile/07193759050963768511noreply@blogger.com4tag:blogger.com,1999:blog-35058937.post-38855355139830818632008-11-17T21:58:00.002-08:002008-11-19T00:52:33.854-08:00Grails + Quartz + Terracotta1) Grails <a href="http://grails.org/Terracotta+Plugin">recently added plug-in support for Terracotta</a>.<br /><br />2) Grails <a href="http://grails.org/Quartz+plugin">recently added plug-in support for Quartz</a>.<br /><br />3) <a href="http://forge.terracotta.org/releases/projects/tim-quartz/">Terracotta supports Quartz</a><br /><br />So....wouldn't it be possible to demonstrate Grails, Quartz and Terracotta all working together? Seems like a fun project.Taylorhttp://www.blogger.com/profile/07193759050963768511noreply@blogger.com1tag:blogger.com,1999:blog-35058937.post-22105370513386557072008-09-21T15:59:00.000-07:002008-09-21T16:50:47.489-07:00What is a Memoizer and why should you care about it?A Memoizer is a class created by <a href="http://g.oswego.edu/">Doug Lea</a>/<a href="http://www.briangoetz.com/">Brian Goetz</a>. It's basically a create-and-store cache. That is, it has one very useful property:<ul><br /><li>A Memoizer is a Generic class that encapsulates the process of creating a value, and remembering the value (memoizing it) so that subsequent calls to get the value will return the memoized value, and not call the Factory method to create it.<br /></ul><br />Why do you care?<ul><br /><li>A Memoizer encapsulates the process of "getOrCreate()" which is highly useful when you are getting a resource that does not change by a key (think cache) and which is generally considered to be expensive to create. <br /></ul><br />But, you ask, I already can do that, why do I need a Memoizer? Glad you should ask. Memoizer is unique in its solution, because it caches a value by key, but its implementation is very efficient. It has the following additional properties:<ol><br /><li>Calls to get items are concurrent</li><br /><li>Calls to get items for a particular key are guaranteed to call the Factory method for that key once and only once</li><br /></ol><br />It's important to understand that what the Memoizer does is both efficient, and correct. It's relatively easy to roll your own "<code>getOrCreate()</code>" method that has only one of those properties (and most people do). It's not so easy - or obvious - to do both and that's why you should use a Memoizer - and not roll your own.<br /><br />To review, let's see the strategy most people would use for calling the Factory method once and only once. Here's how to call the Factory method once and only once:<br /><pre name="code" class="java"><br />/*** SUB OPTIMAL IMPLEMENTATION THAT IS CORRECT, BUT NOT CONCURRENT ***/<br />private final Factory<A,V> factory = ...;<br />private final Map<A, V> map = new HashMap<A, V>();<br /><br />public V getOrCreate(A key) {<br /> synchronized (map) {<br /> if (map.contains(key)) { return map.get(key); }<br /> <br /> // create<br /> V value = factory.create(key);<br /> map.put(key, value);<br /> return value;<br /> }<br />}<br /></pre><br />But of course, the line <code>synchronized (map)</code> should be a tip-off that this implementation is not concurrent <br />since there is only a single lock protecting access to our map. Can we do better? Sure. Let's use a ConcurrentHashMap to make our <code>getOrCreate()</code> method concurrent (but as we'll see, will not have the once-and-only-once property):<br /><pre name="code" class="java"><br />/*** SUB OPTIMAL IMPLEMENTATION THAT IS CONCURRENT, BUT NOT CORRECT ***/<br />private final Factory<A,V> factory = ...;<br />private final ConcurrentMap<A, V> map = new ConcurrentHashMap<A, V>();<br /><br />public V getOrCreate(A key) {<br /> if map.contains(key) { return map.get(key); }<br /> <br /> // map doesn't contain key, create one -- note that first writer wins, <br /> // all others just throw away their value<br /> map.putIfAbsent(key, Factory.create(key)); <br /><br /> // return the value that won<br /> return map.get(key);<br />}<br /></pre><br />So, those are the two implementations, each implements either correctness (once and only once) or performance (concurrent puts and gets) but neither accomplishes both simultaneously. Of course, that's the whole point of this article. To introduce you to Memoizer. So how does Memoizer ensure correctness (once and only once) while being simultaneously concurrent?<br /><br />The trick is to <i>delay</i> the call to the Factory create method into the future. We already know how to make the operation concurrent, but making the operation concurrent means that the first writer will win, and all the other writers will lose. So we <i>delay</i> the Factory create call by wrapping that call in a FutureTask. In other words, instead of putting the <i>actual value</i> into the map, we put a <i>Future</i> (which is a wrapper for getting the value sometime in the future) into the Map. <br /><br />(Note: If you have not yet become familiar with the <code><a href="http://java.sun.com/j2se/1.5.0/docs/api/java/util/concurrent/Future.html">Future</a></code> and <code><a href="http://java.sun.com/j2se/1.5.0/docs/api/java/util/concurrent/FutureTask.html">FutureTask</a></code> classes introduced in the JDK 1.5, you should)<br /><br />By putting a Future into the map - and not our actual value - we can move the work of calling the Factory create method into the future - specifically we can move it until <i>after</i> the winner of the race to put the value into the map has been determined. That enables us to call get - which calls create on the Factory - on the one and only Future instance that is contained within the Map.<br /><br />The full code for Memoizer illustrates this trick. Note that the <code>Computable</code> interface specifies a generic "Factory". Also note that I have not been able to find a library or canonical reference for Memoizer. The best I have is here: <a href="http://www.javaspecialists.co.za/archive/Issue125.html">Memoizer.java</a>.<br /><br />Here it is re-created for your convenience (note that I have adjusted it slightly, for example allowing the user to specify the number of segments created in the underlying ConcurrentHashMap):<br /><pre name="code" class="java"><br />public interface Computable<A, V><br />{<br /> V compute(A arg) throws InterruptedException;<br />}<br /><br />public class Memoizer<A, V> implements Computable<A, V><br />{<br /> private final ConcurrentMap<A, Future<V>> cache;<br /> private final Computable<A, V> c;<br /><br /> public Memoizer(Computable<A, V> c)<br /> {<br /> this(c, 16);<br /> }<br /><br /> public Memoizer(Computable<A, V> c, int segments)<br /> {<br /> this.cache = new ConcurrentHashMap<A, Future<V>>();<br /> this.c = c;<br /> }<br /><br /> public V compute(final A arg) throws InterruptedException<br /> {<br /> while (true) {<br /> Future<V> f = cache.get(arg);<br /> if (f == null) {<br /> Callable<V> eval = new Callable<V>() {<br /> public V call() throws InterruptedException {<br /> return c.compute(arg);<br /> }<br /> };<br /> FutureTask<V> ft = new FutureTask<V>(eval);<br /> f = cache.putIfAbsent(arg, ft);<br /> if (f == null) {<br /> f = ft;<br /> ft.run();<br /> }<br /> }<br /> try {<br /> return f.get();<br /> } catch (CancellationException e) {<br /> cache.remove(arg, f);<br /> } catch (ExecutionException e) {<br /> LaunderThrowable.launderThrowable(e);<br /> }<br /> }<br /> }<br />}<br /></pre><br />You will of course need the <code>LaunderThrowable</code> implementation to compile this example. For a full code listing, hop on over to Terracotta, where I show not only a full working example, but demonstrate how it works with Terracotta across two nodes:<br /><br /><a href="http://www.terracotta.org/web/display/orgsite/Recipe?recipe=memoizer">Full Memoizer Example with Terracotta</a>Taylorhttp://www.blogger.com/profile/07193759050963768511noreply@blogger.com9tag:blogger.com,1999:blog-35058937.post-61556925997604688782008-09-19T13:10:00.001-07:002008-09-19T13:41:56.323-07:00What Is Terracotta?There are so many ways to answer this question - of course our <a href="http://www.terracotta.org">website</a> has it's own way. I recently wrote an e-mail to someone who asked me about Terracotta, and I figured why not share it on my blog?<br /><br />Here is my response (updated a bit to be more appropriate for a blog)...<br /><br />Terracotta clusters at the level of the JVM. It is a 100% Java solution. We use the Java API and Memory Model as an abstraction layer into which we inject clustering. <br /><br />There are some specific differences with the way that we implement clustering and the way that others do (in fact, in that regard, Terracotta is an entirely unique solution, I do not believe there is anything else like it). <br /><br />There are two main differences in Terracotta -the programming model and the performance (the two go hand in hand, one cannot be had without the other). <br /><br />For the programmer , Terracotta is injected at the Java level, meaning that programming a "distributed" application with Terracotta is no different than programming a multi-threaded or concurrent application. Terracotta makes use of all of the concurrent facilities built in to the Java language and API so that the definition and operation of those facilities are extended across a cluster - in other words each node that you add to the cluster simply becomes more threads available to your application.<br /><br />Put another way, you program plain POJOs, and Terracotta manages replication services of those POJOs, maintains the identity of those POJOs and provides locking services using either synchronized/wait/notify or java.util.concurrent libraries e.g. ReentrantReadWriteLock - all across the cluster. (Again the model is simply threads in one node are no different than threads in another node - all standard Java operations "just work")<br /><br />Of course this doesn't mean that programming across nodes separated by a network is free. Terracotta doesn't believe that an architect doesn't have to know about that interaction. We do think one has to architect for Terracotta, but we do not believe you should have to *program* to it. The analogy is much like that of the garbage collector - you don't program to the Java Garbage Collector, but you do architect your application around it's presence. <br /><br />From a high level architectural level, Terracotta uses a tiered architecture. All application nodes talk to the Terracotta Server using TCP (never multicast, and never to one another, P2P is provably not scalable to provide coherent locking). The Terracotta Server can be clustered (called the Terracotta Server Array) for availability and scalability. It's a lot like the Database in that regard - the Terracotta Server (Array) is the composer in the symphony, coordinating the actions of the application nodes, and storing the data safely - all the way to disk in fact just like a database (and transparently from the application nodes perspective). When you need more availability or scalability, you just add more Terracotta Servers (no changes to your application are required).<br /><br />The replicated data in Terracotta is 100% coherent across the cluster, and always stored safely to disk. This feature is unique to Terracotta given the performance levels it can achieve, which is the other main difference between Terracotta and all other solutions in the same space.<br /><br />Other clustering solutions in the same space <i>claim</i> to have linear scaling, coherent data, and high performance - all delivered at the same time. That's basically a lie - none of them come even close to delivering all three at the same time (e.g. coherent data is possible but it's really slow, high scale can be achieved, but only for non-coherent data). And no product in the same space delivers the same performance that Terracotta does - it is simply in a class of its own since it is the only product that does not rely on brute force replication techniques such as Java Serialization. Coupled with some really innovative ways to eliminate or reduce network latency for locking, Terracotta provides a solution that can give data coherence guarantees, with amazing performance.<br /><br />What all of this translates into is a solution that is flexible (a programmer can pick and choose his/her own programming stack and domain model), fast (no other clustering solution can send delta updates over the network), and more importantly, manageable. The Terracotta architecture is not an accident - it is an intentional design decision that ensures that managing a Terracotta Cluster is simple and efficient. Just like the proven 3-tier architecture of web application nodes and database servers, Terracotta stores application clustered memory in a well known location - the Terracotta Server Array. Loss of application nodes in a Terracotta Server Cluster, like in a 3-tier application, does not risk data-loss in any way, and likewise, loss of the Terracotta Server process(es) does not jeopardize the data in any way.Taylorhttp://www.blogger.com/profile/07193759050963768511noreply@blogger.com2tag:blogger.com,1999:blog-35058937.post-21508316006828978672008-09-09T10:46:00.000-07:002008-09-09T11:19:45.848-07:00Cluster Deadlocks *ROCK* with TerracottaAm I crazy or what? <br /><br />No not really. You see I just happen to have seen more than one customer run into a cluster deadlock, and it turns out that solving the issue with Terracotta is awesome (actually, Terracotta can automatically detect it in an upcoming version, but shhh don't tell anyone I told you that)<br /><br />It's funny, really, because I have been hearing this dumb idea that somehow clustered deadlocks with Terracotta is actually this really scary thing -- ooooh watch out for that complicated Terracotta thing, it uses LOCKING (oh gosh) and that can lead to CLUSTERED DEADLOCKS. Oh my. (Anyone know where I can get a clustered deadlock costume, it's almost halloween!)<br /><br />Really. It's like a bad rumor I keep hearing over and over again. What do they call that when people try to scare other people with rumors that aren't true .... F.....oh nevermind. Here's why deadlocks truly are better with Terracotta:<br /><br />First, what do we get with Terracotta?<br />- Kill a JVM, release its lock. <br />- Kill a JVM, don't lose your state<br /><br />Why does that matter? Well what do you do when you see a deadlock with a regular Java application? Since it's pretty much hosed, you have to restart it (usually you probably debug the hell out of it first and try to fix the deadlock). But the app is hosed. Unless you happen to have coded a "stateless" app - you've also lost your app state. Bummer :(<br /><br />Well, not so with an app running on Terracotta. First of all, you don't have to kill the whole app. In fact, if you do actually get a clustered deadlock, you just have to kill <span style="font-style:italic;">one half</span> of the deadlock (because the locks are released, get it?) and the other half will actually get to complete it's operation. How do you do that? Well since the app state is highly available, you can kill any node at will. <br /><br />So it's simple to resolve a clustered deadlock with Terracotta - just do a rolling restart of every client JVM. That's it. When you hit one half of the deadlock, and kill that JVM, the lock that the other side of the clustered deadlock wants will be freed, and it will go on its merry way. <br /><br />Now of course, you still need to debug the hell out of your app :). When you fix the app, just update it in place, do another rolling restart, and voila! Fixed deadlock with no downtime.<br /><br />So to summarize, deadlocks with Java:<br />- Have to restart<br />- Lose app state<br />- Downtime BAD<br /><br />Deadlocks with Terracotta (e.g. Clustered Deadlocks):<br />- Rolling restart of application nodes<br />- Preserve application state<br />- No downtime GOODTaylorhttp://www.blogger.com/profile/07193759050963768511noreply@blogger.com15tag:blogger.com,1999:blog-35058937.post-90354437740553941262008-09-08T06:57:00.001-07:002008-09-08T07:11:45.189-07:00How can I make sure Terracotta is wired into my application?<div>Per the title of this blog post, I'm going to show you how to make sure that Terracotta is enabled in your application.</div><div>One of the funny things about Terracotta is that applications typically don't know it's there. This is normally a really good thing - it means you can use the 'ole <a href="http://blog.slickedit.com/?p=124">binary search debug trick</a>. (Just remove Terracotta from one half of the application one debug session at a time to quickly narrow down the issue).</div><div>But of course, in production, we want to make doubly sure Terracotta is actually running - yeah sure the application might be happy as a clam churning out txns, but we want to make sure those txns are getting replicated for high availability, right? :)</div><div>Fortunately, finding out if Terracotta is wired into your app is really easy. We can make use of the fact that most of the regular Java classes are instrumented by default, so we can use reflection to interrogate one of them. I chose <code>String</code> since I think it's a well known class. Here's the code:</div><pre name="code" class="java"><br /> public static final boolean isTCEnabled()<br /> { <br /> try {<br /> String.class.getMethod("__tc_decompress");<br /> return true;<br /> } catch (Exception e) {<br /> return false;<br /> }<br /> }<br /></pre><div>That's it! We just test to see if a special Terracotta method is present, and if so, we know Terracotta is wired in. All you have to do is put that into your application startup somewhere, and complain loudly if the method returns <code>false</code> :)</div>Taylorhttp://www.blogger.com/profile/07193759050963768511noreply@blogger.com1tag:blogger.com,1999:blog-35058937.post-88886406801906050682008-08-20T00:18:00.000-07:002008-08-20T01:08:54.121-07:00Return to civilization...re-launch of Terracotta.orgIt's been a bit since I posted...we've been really busy cranking away on a new site at Terracotta, and <a href="http://www.terracotta.org/">it's finally out</a>.<br /><br />I'm really pleased with the site, I hope our users are too. The goals for the site were:<br /><ol><li>Simple</li><li>Clean</li><li>Professional</li></ol>and of course, useful! :)<br /><br /><h2> What's New</h2><br /><br />Everything, really :). Well, not everything, but a lot. On the graphics side of things, we added a lot of <a href="http://jquery.com/">JQuery</a> magic. Not so much for the magic itself, but to make the user experience more pleasant. Where possible, popup windows and user transitions have been replaced by images that zoom in place, and drop-down panels that help keep the focus where it should be - on the task at hand.<br /><br />Also of note, although it's of little practical use now, I implemented a nice CSS effect for our menus that allows for them to look 3d and stylish but only requires one simple transparent png (no, I don't care about IE6. It's disgusting). It's basically a minor modification of the transparent text effect described here in <a href="http://www.digital-web.com/articles/web_standards_creativity_png/">this blog post</a>. I'll probably write this up as a separate post.<br /><br />But really, we didn't focus that much on the look or the feel, but the design. The look and feel came primarily from the design, and the goals, so we knew when something worked, and when it didn't. If it was distracting, complicated, or busy, it didn't make the cut. I bet I'll be blogging about the design process before long.<br /><br />Here's a brief preview of some of the new design elements:<br /><br /><h2>Clean Simple Menu</h2><br /><br />Pretty self evident I think.<br /><br /><img style="margin:0 10px 10px 0;cursor:pointer; cursor:hand;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgMAIIuWHhxJF_g6eFkBzqNvisWx5Bj0cORg6Pj-5LHu_2aPOcIqvBThqKHw552ptwGry8U2ffyPzu6fBIktf-ncyHM8Rz_GAB0fKO7uIYNae6vI1v5Yxy_smzSy33bqNbVvAkT/s400/Picture+8.png" border="0" alt="" id="BLOGGER_PHOTO_ID_5236500237101608754" /><br /><br /><h2>Process Oriented Site Flow</h2><br /><br />Well, pretty hard to miss, really. I hope it doesn't get any easier than 1) 2) 3) 4). <br /><br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjsxu4FThXQRzguss4xYojkgMn33xCl1Pf_gAnZkKRsvElPUbloaAsolFvtyDymahQjnU3ayBwe9GEijnJjElJsEZirRGgdBhWFbm3dF8YSD7QpWPJ_uWHbc4BTLSMNX_c9OqnV/s1600-h/Picture+10.png"><img style="margin:0 10px 10px 0;cursor:pointer; cursor:hand;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjsxu4FThXQRzguss4xYojkgMn33xCl1Pf_gAnZkKRsvElPUbloaAsolFvtyDymahQjnU3ayBwe9GEijnJjElJsEZirRGgdBhWFbm3dF8YSD7QpWPJ_uWHbc4BTLSMNX_c9OqnV/s320/Picture+10.png" border="0" alt=""id="BLOGGER_PHOTO_ID_5236500894652149378" /></a><br /><br /><h2>Simple Controls, My Terracotta</h2><br /><br />Main controls are on every page, easy to find, but hopefully unobtrusive until you need them. Also this is the preview of "My Terracotta" - expect more.<br /><br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgHSab6KEPICIY9O8_6N0joINMslQXeYUXL8rWRQp0tSmjvyoUJ4OUr1J5CzJbiR6GJCdVNluLeyBB93ohK85ot301HAxSJeKgDWASW5jJAFNqxxrpUuxhkDYNAcwN_DyRHjBA4/s1600-h/Picture+11.png"><img style="cursor:pointer; cursor:hand;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgHSab6KEPICIY9O8_6N0joINMslQXeYUXL8rWRQp0tSmjvyoUJ4OUr1J5CzJbiR6GJCdVNluLeyBB93ohK85ot301HAxSJeKgDWASW5jJAFNqxxrpUuxhkDYNAcwN_DyRHjBA4/s400/Picture+11.png" border="0" alt=""id="BLOGGER_PHOTO_ID_5236501469342944434" /></a><br /><br /><h2>Drop Down Panels</h2><br /><br />As I mentioned, drop down panels help get stuff done without leaving the context. I think a site shouldn't need Help, but we added it anyway. We all really hope it's unnecessary, but if it helps just one person, it's worth it. We're really committed to getting everyone successful with Terracotta.<br /><br /><br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhnB-_v6EMls85oIpN-eSIj2aVHjzFIT4mWJ_g4p8Z0EoV2ywIzB7VilQzAG27KlNwNThjOfkFXIVjHR5V-YjBnVenNbc4UTD79kjIUTxneJq08DbMExrvnc1GlWdAe_ezLS_Ty/s1600-h/Picture+12.png"><img style="cursor:pointer; cursor:hand;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhnB-_v6EMls85oIpN-eSIj2aVHjzFIT4mWJ_g4p8Z0EoV2ywIzB7VilQzAG27KlNwNThjOfkFXIVjHR5V-YjBnVenNbc4UTD79kjIUTxneJq08DbMExrvnc1GlWdAe_ezLS_Ty/s320/Picture+12.png" border="0" alt=""id="BLOGGER_PHOTO_ID_5236502182977378210" /></a><br /><br /><h2>Lots of new content</h2><br /><br />As I said, we worked really hard on trying to capture how someone should come to Terracotta, learn and understand it, integrate, test, tune, deploy and operate it. The design and the process are integral to the site - so much so that we even embedded some process diagrams to anchor where the user is in the site. <br /><br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiveQFJL2jypzT0iyXxF9JQGXswRqI_F_xCU9yZ6PCy6UY-F8mY0ALxfUsP-SrcaWp8JVRLKRSeLT_8Egt6vV649j9frbRXq-cxqB1HsFBhTJ_pwT191ufy_Porvgs39WbKoaW2/s1600-h/Picture+13.png"><img style="cursor:pointer; cursor:hand;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiveQFJL2jypzT0iyXxF9JQGXswRqI_F_xCU9yZ6PCy6UY-F8mY0ALxfUsP-SrcaWp8JVRLKRSeLT_8Egt6vV649j9frbRXq-cxqB1HsFBhTJ_pwT191ufy_Porvgs39WbKoaW2/s320/Picture+13.png" border="0" alt=""id="BLOGGER_PHOTO_ID_5236502842389793106" /></a><br /><br /><h2>Architecture Patterns</h2><br /><br />Finally, we put some massive effort into capturing and describing successful architecture patterns, and how they work with Terracotta. There's a whole section devoted to describing these patterns, and there is a lot more on the way.<br /><br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiz7a1CtOo4-JUH8w7wwM7z-sxXCmYD9zBATDow7BehyphenhyphenW8VJpHC31ZpLcFaxjRtZPUMsePWmYUxyRhPiN3aymt_xRBimv-QW268msOxSDTyZIpIRmtxCJM7lkOa06md-aVCdnG4/s1600-h/Picture+15.png"><img style="cursor:pointer; cursor:hand;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiz7a1CtOo4-JUH8w7wwM7z-sxXCmYD9zBATDow7BehyphenhyphenW8VJpHC31ZpLcFaxjRtZPUMsePWmYUxyRhPiN3aymt_xRBimv-QW268msOxSDTyZIpIRmtxCJM7lkOa06md-aVCdnG4/s320/Picture+15.png" border="0" alt=""id="BLOGGER_PHOTO_ID_5236503366742650482" /></a><br /><br />Go check it out for yourself, we're live at <a href="http://www.terracotta.org">http://www.terracotta.org</a>. We'd love to hear your feedback.Taylorhttp://www.blogger.com/profile/07193759050963768511noreply@blogger.com3tag:blogger.com,1999:blog-35058937.post-20019083840361494622008-07-14T11:36:00.001-07:002008-07-14T11:38:45.099-07:00Shortcuts using JIRAAt <a href="http://www.terracotta.org">Terracotta</a>, we use <a href="http://www.atlassian.com/software/jira/">JIRA</a> for issue tracking (see here at <a href="http://jira.terracotta.org">jira.terracotta.org</a>).<br /><br />Today, I stumbled on a really nice feature. On a whim, I thought "I wonder if they implement shortcut (hotkeys)?". So I tried it out - sure enough "Ctrl+e" edits a JIRA issue, and "Ctrl+s" saves it. NICE!<br /><br />+1 for Atlassian.Taylorhttp://www.blogger.com/profile/07193759050963768511noreply@blogger.com2tag:blogger.com,1999:blog-35058937.post-35820682164191284502008-03-31T08:57:00.000-07:002008-04-06T09:48:45.059-07:00Chronicles of a Terracotta Integration: CompassLast week, I met up with <a href="http://www.kimchy.org/">Shay Banon</a>, author of <a href="http://www.opensymphony.com/compass/content/about.html">Compass</a>, at the <a href="http://javasymposium.techtarget.com/lasvegas/index.html">The Server Side: Vegas</a> conference. We thought it would be great to see if we couldn't crank out an integration between Terracotta and Compass. You can read more about our integration from <a href="http://www.kimchy.org/compasslucene-terracotta-integration/">Shay himself</a>.<br /><br />I wanted to write a log of our efforts, because I thought it might provide some insight for anyone considering integrating <a href="http://terracotta.org/">Terracotta</a> into their own project. I was particularly happy with our effort, because it outlines what I feel is the best approach for developing with Terracotta. The approach is actually quite simple. Because Terracotta adds clustering concerns to your application using configuration, you don't write code directly to Terracotta. Instead, you just write a simple POJO application <span class="Apple-style-span" style="font-style: italic;">without</span> Terracotta, and then add the clustering later. <br /><br />So the approach I recommend is the following:<ol><li>Figure out how to implement the solution using a single JVM. NO TERRACOTTA. Use just simple POJOs and threads.</li><br /><li>Implement and test your solution.</li><br /><li>It helps to have envisioned, beforehand, what part of your implementation will become a Terracotta root. But it's not necessary. If your application is stateful, it will have a root.</li><br /><li>Using the root, start with a basic Terracotta config file, <a href="http://www.terracotta.org/confluence/display/docs1/Configuring+Terracotta">and build up the appropriate config file to cover all the instrumentation and locking</a>.</li><br /><li>Test your application again, with a single jvm, but this time with Terracotta.</li><br /><li>Tune your implementation.</li><br /><li>Move to 2 or more JVMs.</li><br /></ol>That's it. So how did this play out for the Compass integration? Here is my rough recollection of the action.<br /><br /><span style="font-weight: bold;">10:00 am - Shook Hands</span> - Shay and I met up at the conference.<br /><br /><span style="font-weight: bold;">10:05 am - Started coding</span> - First we chatted a bit about our strategy. It seemed easiest to start with the existing Lucene RAMDirectory implementation and tune it up a bit.<br /><br /><span style="font-weight: bold;">10:30 am - Strategy decided</span> - Based on my knowledge of Terracotta, and Shay's knowledge of Lucene/Compass, we decided on the following:<ul><li>Start with the Lucene RAMDirectory implementation, but rewrite it as necessary to fit a simple POJO model</li><br /><li>Since RAMDirectory is mostly unmaintained, we knew we had to just go through the implementation and clean it up. It comprises about 4 classes total, about 100 lines long, so the task was feasible.</li><br /><li>Because Terracotta can just "plug" in to a well written application, and Shay has a comprehensive unit test suite (over 1,000 tests), a load test, and a concurrency test, we'd write the implementation first in POJOs</li><br /><li>After verifying that the implementation works as expected in pure POJOs, then we would work on the configuration to inject Terracotta clustering</li><br /><li>After running the solution with Terracotta, we would tune it.</li><br /><li>And finally, we would wrap up various bits and pieces into a Terracotta Integration Module (TIM)</li></ul><span style="font-weight: bold;">11:30 am - POJO Implementation done</span> - We ended up rewriting the RAMDirectory, which was fine because it was in need of an overhaul anyway. Rewriting its implementation meant we now had a good understanding of the implementation.<blockquote>Just a quick note - it was a real joy coding with Shay. He is a super smart guy, and it's great to work with someone like that. Of note, he really understands synchronization, which is really important to write applications correctly. Even better, he really got the principle of writing better code by writing less code. We went through the RAMDirectory implementation with a weed wacker, and what was left was about 1/2 the code. That was more readable and more maintainable. And is better performing. That was fun.</blockquote><span style="font-weight: bold;">12:00 pm - Unit Tests pass</span> - With some minor corrections, we had unit tests passing. We were both running out of power, and hungry, so we took a break to eat lunch, and agreed to resume in the afternoon.<br /><br /><span style="font-weight: bold;">1:30 pm - Write the Terracotta config file</span> - While writing the POJO implementation, we already knew the key concepts we were going to need for writing up our config file. We added the appropriate instrumentation. We added the locking. A few config statements later, we had a working Terracotta configuration.<br /><br /><span style="font-weight: bold;">2:00 pm - We had Compass running on Terracotta!</span> - Approx. time elapsed? 2 1/2 hours (most of which was spent rewriting the RAMDirectory implementation)<br /><br /><span style="font-weight: bold;">2:30 pm - Tuning Time</span>- At first Shay threw me - he said oh man it looks like it's running really fast. Except it turns out he wasn't testing the right thing. And then he tells me oh man its really slow!<br /><br />Now don't misunderstand this. I know Terracotta can go really fast. But I wasn't in the least bit surprised. And you shouldn't be either. How many pieces of code have you ever written that compiled and ran correctly - on the first try? Right. One, if you are lucky.<br /><br />Terracotta is kind of like that. The first step is to get it right. And that means synchronization, and locking, and once you have all that, your application runs correctly, but slowly.<br /><br />Fortunately, <span style="font-weight: bold;">it's easy to fix</span>.<br /><br />And so I taught Shay how to tune up his Terracotta integration. Or rather, I showed him the tools he needed, and he went to town. I just sort of stood by watching, giving the occasional comment or two.<br /><br />This was the fun part. It was time to take out the Admin console. The Terracotta Admin console gives you a wealth of information about your application. Of note:<ul><li>You can browse your clustered data in realtime</li><br /><li>You can monitor realtime statistics - including Terracotta txns/sec, Java Heap Memory, and CPU</li><br /><li>You can access lock statistics using the lock profiler</li><br /><li>You can snapshot over 30 metrics using the Statistics Recorder <span style="font-style: italic;">and visualize them using the Snapshot Visualizer</span></li></ul>We started first with the object browser. Once convinced that we had the right data in the cluster, we moved on to performance.<br /><br />On our first run, we measured the Terracotta txns/sec. I was actually pretty impressed to see that our server on his MacBook Pro was cranking out 10k/sec. But I knew we wanted this number to be <span style="font-style: italic;">lower</span>. A lot lower.<br /><br />So here comes the <span style="font-weight: bold;">first</span> rule for tuning Terracotta: adjust your locking to match your workload. It turns out that we had enabled an autolock for every single byte being written to the Lucene "files" - and this was hurting us pretty bad. Because we already had a lock on our byte array that we were writing to, we actually just deleted the synchronization, and the lock config from the method that wrote bytes into the "file" - and we observed a big drop in the Terracotta txns/sec. We went from the aforementioned 10k/sec to about 1750/sec.<br /><br />Now what this means is that the Terracotta server was working just about 10x less for the same workload. And that means we were doing more work/transaction, and so our performance improved accordingly. You get the same effect with Hibernate - it batches up a bunch of little POJO updates into a single SQL statement - and that means you can do more <span style="font-style: italic;">real</span> work because each SQL statement has more data in it. Lots of little SQL statements means lots of overhead, and maybe more SQL queries executed/sec, but much less application txns/sec. Same concept here with locking.<br /><br />How did we identify what lock(s) to target? <br /><br />That's the <span style="font-weight: bold;">second</span> rule of tuning with Terracotta: USE THE ADMIN CONSOLE<br /><br />We used the lock profiler feature included in the Admin Console to determine the exact stack trace that generated these locks. The process is simple:<ul><br /><li>Enable lock profiling with stack traces in the Admin Console, </li><br /><li>run your application,</li><br /><li>then refresh the view to get a count of the lock acquires/releases/held times etc.</li><br /><li>sort on # of lock acquires, and now you know what lock is being requested the most, what stack trace caused that lock, and what Terracotta config was responsible for making that lock.</li></ul>Armed with this knowledge, Shay set about eliminating most of our superfluous locking. Turns out that creating a Lucene "file" is a single threaded affair, so were able to create a single lock to cover the entire process of "writing" to a file, and that cut out about 90% of our locking.<br /><br />At the end we got down to about 750 Terracotta txns/sec, which improved the application performance quite a bit.<br /><br />Still not satisfied, we moved on to the Terracotta Statistics Recorder. This is a new feature in Terracotta 2.6.<br /><br />Turning on this feature records just about everything you ever wanted to know about your application, Terracotta, the JVM, and your system (including CPU, disk stats, and network stats). You can export these stats as a CSV file, and import them into our <a href="http://www.terracotta.org/confluence/display/orgsite/Get+Snapshot+Visualization+Tool">Snapshot Visualizer Tool</a>. The SVT gives you a view like so:<br /><img style="margin:0 10px 10px 0;cursor:pointer; cursor:hand;width: 400px;" src="http://wiki.terracotta.org/confluence/download/attachments/9142540/Picture%202.png" border="0" alt="" /><br /><br /><span style="font-weight: bold;">4:30 pm - TIM time</span> - We were pretty satisfied with the performance. Even though we wanted more, Shay felt it was best to focus on turning Compass into a TIM (Terracotta Integration Module).<br /><br /><span style="font-weight: bold;">5:30 pm - Time to call it quits</span> - We had hacked up the ant build.xml file to get ourselves a TIM in no-time - except that it wouldn't quite load correctly. (Later we learned we had just specified the filename wrong - easy fix).<br /><br />Overall, I thought we had a pretty good day. We wrote and tuned a Terracotta integration in about 6 hours flat. With a few more hours of work, <a href="http://www.kimchy.org/compasslucene-terracotta-integration/">Shay was able to complete the integration</a>.<br /><br />I was really happy to use some of the recent tools we have been building, like the Lock Profiler and the Statistics Recorder. Seeing the real-world use of those was invaluable, and confirmed that our commitment to enabling the developer to self-tune by providing enhanced visibility is spot on.<br /><br />I am looking forward to people <a href="http://www.terracotta.org/confluence/display/orgsite/Download">downloading 2.6</a>, trying out these awesome tools for themselves and <a href="http://www.terracotta.org/confluence/display/orgsite/Get+Snapshot+Visualization+Tool">providing feedback</a>!Taylorhttp://www.blogger.com/profile/07193759050963768511noreply@blogger.com9tag:blogger.com,1999:blog-35058937.post-67803038699625129892008-03-30T20:55:00.000-07:002008-04-16T07:17:30.775-07:00Fun with Distributed ProgrammingSomething about the nature of distributed programming makes it quite divisive. You either love it or despise it. It's rare that I've run into someone who is ambivalent about it.<br /><br />Those that love it, love it because it's hard core. They're proud to know all the ins and outs of dealing with failures, at the system, network, and application level. All of that specialized knowledge is also what turns off the rest of us.<br /><br />It's kind of like database programming. There are only a select few who really like it. The rest of us only do it because we have to.<br /><br />Well, I honestly think that Terracotta changes the game. The key is that Terracotta makes distributed programming <span style="font-style: italic;">fun</span> because it takes away most of the distributed programming part, leaving you with just the <span style="font-style: italic;">fun</span> part.<br /><br />It reminds me of when Linux came out. Everyone loved it because they could just tinker with different schedulers, and not have to think about building an OS from scratch, just to try out a new idea. That's what Terracotta is like. It manages all the hard networking and distributed programming parts, so you get to just play with the algorithms.<br /><br />Interested? Let's look at a real (if contrived) example. Let's suppose that you have to build the following:<br /><ul><br /><li>a service that executes periodically to do some work</li><br /><li>you don't care where this service runs, only that it runs</li><br /><li>it <span style="font-style: italic;">has</span> to run, but one and only one system can run it</li><br /><li>you've got a cluster of n systems, you'd like any one of them to be responsible for running the service</li><br /></ul>If it were a single JVM, you could do a thousand things, like use a java.util.Timer, or Quartz, or even your own simple Thread with a delay loop in it.<br /><br />But in a cluster? The choices for synchronizing the behavior of a number of JVMs across a cluster quickly eliminate the <span style="font-style: italic;">fun</span> part, leaving just tedious, boring, and mundane work to be done. Cluster synchronization you're thinking. What should I use? TCP? Multicast? Shared file system locking? A shared database? RMI? JMS? EJBs? Oh dear.<br /><br />But wait. Terracotta provides synchronization primitives that work across the cluster just like in a single JVM. So that means getting this right in a single JVM means getting it right across the cluster. Could it really be that easy? And <span style="font-style: italic;">fun</span>? Yes!<br /><br />Let's have a look. For the sake of simplicity, let's do the simple thing. We'll write the delay loop version. We'll implement it as a Singleton that implements Runnable, so we can pass the Singleton to a Thread. Here it is:<pre name="code" class="java">public class SimpleWorkRunner implements Runnable<br />{<br /> // mark as a Terracotta root<br /> private static SimpleWorkRunner singleton = new SimpleWorkRunner();<br /><br /> // singleton pattern - private constructor so there is only one<br /> private SimpleWorkRunner() { }<br /><br /> public static SimpleWorkRunner instance() { return singleton; }<br /><br /> public synchronized void run()<br /> {<br /> while (true) {<br /> // do work<br /> ...<br /> try { Thread.sleep(2000); } catch (InterruptedException e) { }<br /> }<br /> }<br />}</pre><br />That's it! In every JVM, kick off a new thread against the singleton:<pre><br />...<br />new Thread(SimpleWorkRunner.instance()).start();<br /></pre><br />And we're done!<br /><br />You might have noticed one thing - the run method is synchronized. In a single JVM, this will mean that more than one Thread executed against this Singleton will result in only one Thread winning the synchronization race, and executing the run() method. <br /><br />In a single JVM, this may not be that important, since there might only ever be one Thread. But with more than one JVM, we will always start at least one Thread per JVM, and that means we have to ensure, per our requirements, that only one Thread ever enters run() at a time.<br /><br />Terracotta takes care of that for us. We just write the synchronized block, and Terracotta converts that into a cluster lock. And just like in a single JVM, only one Thread - across the cluster - will win the race to enter the run() method.<br /><br />(Also of note is that this particular implementation assumes that one and only one Thread should assume control and never relinquish it. That was the purpose of the implementation, if you wanted to "bounce" the control around the cluster then we should implement the run method differently depending on the requirements.)<br /><br />The Terracotta config for this class is trivial. We need to tell Terracotta that the singleton should be a Terracotta root. A Terracotta root will always be the same instance across the entire cluster, which is exactly what we want for a singleton. And we need to autolock the run method so the synchronization is applied to the cluster, not just a local JVM. Here's the config for that:<pre name="code" class="xml"><tc:tc-config xmlns:tc="http://www.terracotta.org/config"<br />xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"<br />xsi:schemaLocation="http://www.terracotta.org/schema/terracotta-4.xsd"><br /><br /> <application><br /> <dso><br /> <locks><br /> <autolock><br /> <method-expression>void SimpleWorkRunner.run(..)</method-expression><br /> </autolock><br /> </locks><br /> <roots><br /> <root><br /> <field-name>SimpleWorkRunner.instance</field-name><br /> </root><br /> </roots><br /> </dso><br /> </application><br /></tc:tc-config></pre><br />We didn't have to worry about the dirty details. Teracotta did. And that means distributed programming becomes <span style="font-style: italic;">fun</span> again!<br /><br />Find out more:<br /><ul><li><a href="http://www.terracotta.org/">Terracotta.org</a> - home page</li><li><a href="http://www.terracotta.org/confluence/display/orgsite/Start+Using+Terracotta">Quick Start</a> - download and see the demos<br /></li><li><a href="http://www.terracotta.org/confluence/display/howto/Cookbook">Cookbook</a> - simple recipes that demonstrate Terracotta in action<br /></li></ul><br />Note that this example is very similar to the <a href="http://www.terracotta.org/confluence/display/howto/Recipe?recipe=singleresource">Single Resource</a> recipe. Try it out first to get started.<br /><br /><b>Extra Credit</b><br />How does another JVM in the cluster gain control of the task? (Hint: Is it possible for more than one Thread to enter the critical section in <code>run()</code>? In normal Java - no. But what happens in Terracotta with more than one JVM?)Taylorhttp://www.blogger.com/profile/07193759050963768511noreply@blogger.com7tag:blogger.com,1999:blog-35058937.post-27612897340002295152008-03-22T15:56:00.000-07:002008-07-04T11:45:51.922-07:00A Clustered ClassLoaderIf you're building a distributed system, or contemplating building a distributed system, you might have run into this one before:<br /><ol><br /> <li> You write and compile your classes in Eclipse<br /> <li> You try out your classes on your laptop -- they work (woohoo!)<br /> <li> Its a distributed system so you need to make sure your classes work in a true distributed environment<br /> <li> You publish your classes to the distributed systems<br /> <li> You try out your classes -- and they don't work (boo!)<br /> <li> You fix the problem.<br /> <li> You publish the classes again.<br /> <li> Rinse, repeat.<br /></ol><br /><br />After doing this a few dozen times, you find that publishing your classes to distributed systems is a total <a href="http://www.urbandictionary.com/define.php?term=PITA">PITA</a> that you would rather avoid altogether.<br /><br />Or, you might have an application, like <a href="http://www.infoq.com/articles/master-worker-terracotta">Master/Worker</a> in which you deploy some part of the application at deploy time, but you deploy other parts of it during run-time. In the Master/Worker case, you deploy the Master and the Worker, but the Work comes and goes, and you'd like to be able to deploy new Work easily and trivially. In the Master/Worker case, since Masters are usually in control, and there is a farm of Workers, you'd like to deploy some new work to the Master, and let it send the Work to the Workers. Knowing about the Work up front on the Workers is a non-starter.<br /><br />Some solutions to this problem?<br /><ul><br /><li>Java has dynamic code loading capabilities already. Deploy your class files to a shared filesystem like NFS, and deploy your code to a shared directory. <br /><li>Java also supports loading code from URLs (thanks to it's Applet heritage) so deploy your code to an HTTP server and you're set<br /><li>Factor your application such that new classes aren't needed - just make the new definitions "data" driven<br /><li>Embed a scripting engine, so you can pass Strings and interpret them as code - BeanShell, Jython, JRuby, Javascript, and Groovy all come to mind here...<br /></ul><br /><br />Those are all fine solutions, but it never hurts to have more tools in your toolbox does it? Especially if you're already using <a href="http://www.terracotta.org">Terracotta</a>, wouldn't it be nice if there was some way to just leverage Terracotta's core clustering capabilities to build a clustered classloader?<br /><br />I've done just that. Here's how it works:<br /><ul><br /><li>Your application tries to instantiate a class, which means it asks the currently in scope ClassLoader to instantiate the class (by name)<br /><li> By launching the application under the clustered classloader, it is in scope.<br /><li> The clustered class loader has a <code>Map<String, byte[]></code> that correlates classnames to bytes<br /><li> The clustered class loader looks in this <code>Map</code>, if the classname is found, it uses the <code>byte[]</code> to create the requested class using <code><a href="">defineClass()</code></a><br /><li> If the class wasn't found in the <code>Map</code>, then it looks in the filesystem to find the class<br /><li> If the class bytes are found on the filesystem, then it reads them into a <code>byte[]</code>, and stashes them in the clustered <code>Map<String, byte[]></code><br /><li> If the bytes aren't found, it just delegates to the parent classloader<br /></ul><br /><br />I've omitted some of the finer details. The <code>Map</code> used is actually a <code>Map<String, ClassMetaData> </code> where <code>ClassMetaData</code> is a class that holds a <code>long modified</code> and <code>byte[] bytes</code>.<br /><br />Let's have a look at the important parts of the ClusterClassLoader:<pre name="code" class="java"><br />public class ClusterClassLoader extends ClassLoader<br />{<br /> private static final String NAME = "ClusterClassLoader";<br /> <br /> private static Map<String, Class> classes = new HashMap<String, Class>();<br /> private static Map<String, ClassMetaData> bytes = new HashMap<String, ClassMetaData>();<br /> private static transient boolean loaded;<br /><br /> ...<br /></pre><code>ClusterClassLoader</code> is defined to extend <code>ClassLoader</code>. It has a <code>NAME</code> field, which will be used to give a name to this classloader. This is a requirement for a classloader used by Terracotta. Normally Terracotta does this for you, but we are defining a new classloader, so we have to follow the naming rules for Terracotta (naming gives ClassLoaders across the cluster a unique identity).<br /><br />A <code>classes</code> field is defined, which caches the result of the <code>defineClass</code> operation in the local JVM only. A <code>bytes</code> field is defined. This field is marked as a root, so that it can be shared with every other instance of ClusterClassLoader in the cluster.<br /><br />The constructor detects if Terracotta is loaded using some reflection, and if so registers the classloader and sets a flag to enable cluster classloading features:<pre name="code" class="java"> public ClusterClassLoader()<br /> {<br /> super(ClusterClassLoader.class.getClassLoader());<br /> try {<br /> Class namedClassLoader = findClass("com.tc.object.loaders.NamedClassLoader");<br /> Class helper = findClass("com.tc.object.bytecode.hook.impl.ClassProcessorHelper");<br /> Method m = helper.getMethod("registerGlobalLoader", new Class[] { namedClassLoader }); <br /> m.invoke(null, new Object[] { this });<br /> loaded = true;<br /> } catch (Exception e) {<br /> // tc is not present, so don't do anything fancy<br /> loaded = false;<br /> }<br /> }<br /></pre><br /><br />Next, the definition of <code>loadClass</code> is overridden:<pre name="code" class="java"> @Override<br /> public Class<?> loadClass(String name) throws ClassNotFoundException<br /> {<br /> return findClass(name);<br /> }<br /></pre>and so is <code>findClass</code>: <pre name="code" class="java"> @Override<br /> protected Class<?> findClass(String name) throws ClassNotFoundException<br /> { <br /> if (!loaded) {<br /> return getParent().loadClass(name); <br /> }<br /> <br /> Class result = null;<br /> synchronized (classes) {<br /> result = classes.get(name);<br /> if (result != null) { return result; }<br /><br /> result = loadClassBytes(name);<br /> if (result == null) { return getParent().loadClass(name); }<br /> classes.put(name, result);<br /> }<br /> <br /> return result;<br /> }<br /></pre>This is the bulk of the algorithm. The loaded flag is set when the class loader is instantiated. It used a bit of reflection to determine if Terracotta was even present in the JVM. If not, it is set to false, and the ClusterClassLoader just delegates to the parent class loader.<br /><br />If Terracotta is present, then it checks to see if the class has already been defined. If so, it is returned directly from the classes cache. If it has not, then it gets the bytes from the loadClassBytes method. If that cannot find the bytes, then it asks the parent class loader to load the class.<br /><br />The bulk of the implementation is done in the <code>loadClassBytes</code> method:<pre name="code" class="java"><br /> private Class loadClassBytes(String name) throws ClassNotFoundException<br /> {<br /> ClassMetaData metaData;<br /> <br /> synchronized (bytes) { <br /> try {<br /> File f = null;<br /> metaData = bytes.get(name);<br /> URL resource = ClassLoader.getSystemResource(name.replace('.',File.separatorChar)+".class");<br /> // if resource is non null, then the class is on the local fs (in the cp)<br /> if (resource != null) {<br /> f = new File(resource.getFile()); <br /> }<br /> <br /> if (metaData != null) {<br /> // if it's cached, but not on the fs, return it.<br /> // if it's cached, but on the fs, check to see if it's <br /> // up to date<br /> if (f == null || metaData.modified >= f.lastModified()) { <br /> return defineClass(name, metaData.bytes, 0, metaData.bytes.length, null);<br /> }<br /> }<br /> <br /> // load from the fs<br /> byte[] classBytes = loadClassData(f);<br /> Class result = defineClass(name, classBytes, 0, classBytes.length, null);<br /><br /> try {<br /> result.getDeclaredField("$__tc_MANAGED");<br /> // it's managed so cache it<br /> bytes.put(name, new ClassMetaData(f.lastModified(), classBytes));<br /> } catch (NoSuchFieldException e) {<br /> // not managed don't cache it<br /> }<br /> return result;<br /> } catch (IOException e){<br /> return null;<br /> } <br /> }<br /> } <br /></pre>This method looks for the cached bytes, and for a file that corresponds to the class. If both are found, then it compares the modified date of the two. If the modified date of the bytes are greater than or equal to the file, then it returns the bytes in the cache. Otherwise it loads the bytes from the file. Once the bytes are loaded, <code>defineClass</code> is called to turn the bytes into a class file. <br /><br />At this point, the ClusterClassLoader can check to see if the class is instrumented by Terracotta. Every instance of a class that is shared by Terracotta must be instrumented, so it's not necessary to cache class bytes for classes that are not instrumented by Terracotta. If the class is instrumented by Terracotta, then the ClusteredClassLoader stashes the bytes into the class bytes cache.<br /><br /><a href="http://svn.terracotta.org/svn/forge/projects/labs/tim-clusterclassloader/src/main/java/org/terracotta/modules/clusterclassloader/ClusterClassLoader.java">Click here if you would like to see the source code to ClusterClassLoader in its entirety</a><br /><br /><b>UPDATE: This project has been included in the tim-tclib project, and is a runnable sample. <a href="http://svn.terracotta.org/svn/forge/projects/labs/tim-tclib/trunk/samples/clusterclassloader/readme.html">More details can be found in the sample readme.html</a></b><br /><br />I've put the whole thing together as a simple runnable example. You just have to check out the source for the project, and run a few simple Maven commands. You can get the demo from here:<pre><br />$ svn checkout http://svn.terracotta.org/svn/forge/projects/labs/tim-clusterclassloader clusterclassloader<br />$ cd clusterclassloader</pre><br /><br />The demo defines a main project, and two sub projects. The first sub project, sample, is responsible for putting classes into a queue. The second sub project, sample2, reads from the queue. To test the effectiveness of the cluster class loader, the second sample of course does not have the classes from the first sub project.<br /><br />To run the demo:<br /><ol><br /><li>Build the project:<pre><br />$ mvn install</pre><li>Cd to the sample directory, compile and start a tc server:<pre><br />$ cd sample<br />$ mvn package<br />$ mvn tc:start</pre><li> Start the sample process:<pre><br />$ mvn tc:run<br /></pre><li> In another terminal, cd to the sample2 directory:<pre><br />$ cd sample2</pre><li>Compile, and run the example:<pre><br />$ mvn package<br />$ mvn tc:run</pre><br /></ol><br /><br />If you did everything correctly, you should see:<pre><br />[INFO] [node] Waiting for work...<br />[INFO] [node] This is Callable2 calling!<br /></pre>In the second terminal (sample2). The message printed ("This is Callable2 calling!") is printed by a class that is only present in the classpath of the first instance (sample).Taylorhttp://www.blogger.com/profile/07193759050963768511noreply@blogger.com14tag:blogger.com,1999:blog-35058937.post-33442932696253084542008-03-17T20:39:00.001-07:002008-04-01T21:43:45.149-07:00Stupid Simple JVM CoordinationIf you think cross-jvm coordination is easy - then this post is not for you. If it makes you cringe inside, just trying to remember the JMS interfaces, or JGroups api, java.io classes, or figuring out how to mess with a database, then carry on, intrepid reader. This post is for you.<br /><br />I'm going to show you how stupid simple it is to use Terracotta to send a message from one JVM to the other. We'll use two JVMs - a producer and a consumer. I want the producer to create and send a message to the consumer. I want the producer to wait for the consumer to consume the message. When the message is consumed I want the producer to use the return value from the consumer.<br /><br />This would be stupid hard if it weren't for two amazing technologies. The first is the <a href="http://java.sun.com/j2se/1.5.0/docs/api/java/util/concurrent/package-summary.html">java.util.concurrent</a> package. The second is <a href="http://www.terracotta.org/">Terracotta JVM Level Clustering</a>. Putting them together gives you stupid simple JVM coordination.<br /><br />The scenario I outlined is actually ridiculously easy in a single JVM using the java.util.concurrent package. It was built to handle these scenarios and more at the flick of a wrist. Instantiate a queue, fire off a couple of threads, use a <a href="http://java.sun.com/j2se/1.5.0/docs/api/java/util/concurrent/FutureTask.html">FutureTask</a>, and you're done.<br /><br />And you know what? Could it get any more simple than writing one line of code to cluster that queue, and move from two threads in one JVM to one thread in two JVMs. It can't. <br /><br />Here's the main method that does it all:<br /><br /><pre name="code" class="java"><br />public class Main<br />{<br /> public static final Main instance = new Main();<br /><br /> private AtomicInteger counter = new AtomicInteger(0);<br /> private BlockingQueue<FutureTask> queue = new LinkedBlockingQueue<FutureTask>();<br /><br /> public void listen() throws InterruptedException<br /> {<br /> while (true) {<br /> queue.take().run();<br /> }<br /> }<br /><br /> public void run() throws Exception<br /> {<br /> if (counter.getAndIncrement() == 0) {<br /> System.out.println("Waiting...");<br /> listen();<br /> return;<br /> }<br /> <br /> FutureTask task = new FutureTask(new MyCallable());<br /> queue.put(task);<br /> System.out.println("Task completed at: " + task.get().toString());<br /> }<br /><br /> private static class MyCallable implements Callable<br /> {<br /> public Object call() throws InterruptedException<br /> {<br /> System.out.println(new Date().toString() + ": Sleeping 2 seconds...");<br /> Thread.sleep(2000);<br /> System.out.println("Hello world");<br /><br /> return new Date();<br /> }<br /> }<br /><br /> public static void main(String[] args) throws Exception<br /> {<br /> instance.run();<br /> }<br />}<br /></pre><br /><br />And the Terracotta config:<br /><br /><pre name="code" class="xml"><br /><tc:tc-config xmlns:tc="http://www.terracotta.org/config"<br /> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"<br /> xsi:schemaLocation="http://www.terracotta.org/schema/terracotta-4.xsd"><br /><br /> <application><br /> <dso><br /> <instrumented-classes><br /> <include><br /> <class-expression>Main$MyCallable</class-expression><br /> </include><br /> </instrumented-classes><br /> <roots><br /> <root><br /> <field-name>Main.instance</field-name><br /> </root><br /> </roots><br /> </dso><br /> </application><br /></tc:tc-config><br /></pre><br /><br />That's all there is to it. Output looks like this:<br /><br />Node 1:<br /><pre><br />$ javac *.java<br />$ start-tc-server &<br />$ dso-java Main<br />Waiting...<br />(after starting other node...)<br />Mon Mar 17 17:53:45 PDT 2008: Sleeping 2 seconds...<br />Hello world<br /></pre><br /><br />Node 2:<br /><pre><br />$ dso-java Main<br />Task completed at: Mon Mar 17 17:53:47 PDT 2008<br /></pre><br /><br />I've actually written this entire example up as a Recipe on Terracotta.org. Full details and instructions are listed there in the <a href="http://www.terracotta.org/confluence/display/howto/Recipe?recipe=futuretask">FutureTask recipe</a>.Taylorhttp://www.blogger.com/profile/07193759050963768511noreply@blogger.com2tag:blogger.com,1999:blog-35058937.post-50121631356437421582008-03-08T13:33:00.000-08:002008-03-08T13:38:53.777-08:00The Trouble With Data Partitioning (cartoon)<a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEifAojxhUhQfhRIx6uyOdmWib654LxAWSO5f_QvzOEjsp2bmbQbaRqDpT9zZ5RYkJZziYu3R5AxR3REYWQJxrYiTY33qB8OgiQ74whWv3tgvyDrsYyfuAcyk_a89zjODG2HpBYh/s1600-h/Trouble+With+Data+Partitioning.png"><img style="float:left; margin:0 10px 10px 0;cursor:pointer; cursor:hand;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEifAojxhUhQfhRIx6uyOdmWib654LxAWSO5f_QvzOEjsp2bmbQbaRqDpT9zZ5RYkJZziYu3R5AxR3REYWQJxrYiTY33qB8OgiQ74whWv3tgvyDrsYyfuAcyk_a89zjODG2HpBYh/s400/Trouble+With+Data+Partitioning.png" border="0" alt=""id="BLOGGER_PHOTO_ID_5175488421396298994" /></a>Taylorhttp://www.blogger.com/profile/07193759050963768511noreply@blogger.com0tag:blogger.com,1999:blog-35058937.post-53141082528650360462008-03-05T14:22:00.000-08:002008-04-01T21:52:50.843-07:00It's the little things that matter...It's often the little things in a design that make the biggest difference. Sure you have to get the big things right too, but all too often products suffer from a great idea implemented poorly.<br /><br />So at Terracotta, I often have conversations along these very lines. The job we've carved out for ourselves - clustering the entirety of the Java Virtual Machine, is pretty big. That's why it's such a great place to work - the challenge we face is enormous, and it's enormously fun to tackle it. Let me tell you right now, clustering the VM itself isn't going to happen if you don't get the big ideas right. I'll wager that we have, but only history can prove that one right. But just as important is getting the little things right.<br /><br />Today I just happened to discover one of those little things. What is it? Well, if you don't already know, Terracotta maintains <a href="http://blog.terracottatech.com/archive/2005/08/object_identity.html">Object Identity</a> across a cluster of JVMs. That in itself is an amazing feat (no other piece of technology I have ever run into can do this). So Object Identity is the big thing. What's the little thing?<br /><br />Here goes.<br /><br />First, my sample code (Main.java):<pre name="code" class="java">public class Main<br />{<br /> public static final Main instance = new Main();<br /><br /> private Map<Object, Object> map = new HashMap<Object, Object>();<br /><br /> public void run() throws Exception<br /> {<br /> Object key = new Object();<br /> Object value = new Object();<br /><br /> while (true) {<br /> synchronized (map) {<br /> map.put(key, value);<br /> }<br /> Thread.sleep(500);<br /> }<br /> }<br /><br /> public static void main(String[] args) throws Exception<br /> {<br /> instance.run();<br /> }<br />}<br /></pre><br /><br />Those of you not familiar with Terracotta might wonder what's so interesting about this. Well, with Terracotta, you can cluster <i>any</i> Java object, so with the following bit of config, I have done just that:<div><br /><br />tc-config.xml:<pre name="code" class="xml"><tc:tc-config xmlns:tc="http://www.terracotta.org/config"<br />xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"<br />xsi:schemaLocation="http://www.terracotta.org/schema/terracotta-4.xsd"><br /><br /><application><br /> <dso><br /> <locks><br /> <autolock><br /> <method-expression>void Main.run(..) </method-expression><br /> </autolock><br /> </locks><br /> <roots><br /> <root><br /> <field-name>Main.instance</field-name><br /> </root><br /> </roots><br /> </dso><br /></application><br /></tc:tc-config><br /></pre><br /><br />The map in the Main class listed above is now a clustered map (because the root field, <code>instance</code>, holds a reference to it, and therefore transitively it becomes clustered). Anything I put in the map is clustered (transitively again), meaning every object I put in the map is available to all other JVMs in the cluster. That's pretty cool in it's own right (I happen to think), but how is that different from a normal get/put API in a traditional clustered cache, say EHCache, JCS, or OSCache?<br /><br />Well, I monitored the number of transactions the little test above generated. How many would you guess? 1? 100? 1 every 500ms?<br /><br />The answer is actually : 1. Because of Object Identity, after the first iteration through the loop, Terracotta <i>knows</i> that it can optimize out the subsequent calls - there is no need for Terracotta to "re-put" an object for a key that already has that same relationship in the map - so it can save the roundtrip work to the server. <br /><br />In the clustering world, anything you do on the network is orders of magnitude slower than main memory, so every little thing you can do to keep operations local means a big improvement in latency and throughput. So it may be a minor optimization, but it's got a big effect on the latency and throughput of this application. My application may be trivial, but consider if that map was an HTTP Session Context, or a distributed cache. <br /><br />Furthermore, this optimization is simply not possible with serialization based solutions (which must implement a copy on read, copy on write strategy), because it's simply not possible for a serialization based approach to track object identity, or changes to objects, and optimize out this kind of a scenario. <br /><br />However because Terracotta is at the VM level, it knows implicitly when objects change, because of Object Identity, so it is not necessary for a caller of the map to "re-put" objects into the map to make sure it's updated (and it's thus valid to eliminate the subsequent put calls that are superfluous). So in the end - Terracotta would work exactly the same with or without this optimization - the correctness is unaffected by it - but with it, it can, depending on your usage, make your application run orders of magnitude faster. <br /><br />So, in summary, you gotta get the big things right. Object Identity is the big thing. But it's in getting the little things right - for example optimizing away unnecessary network calls by eliminating redundant map.put() calls, that turn out to take a great idea and make it truly impressive.<br /><br /><i>Note that I can't take credit for this, since I had nothing to do with creating the feature or even suggesting it. I just happened to have realized that it's trivial to test to see if it's implemented or not, and I did test it and hoped that you would find the results interesting too.</i><br /><br />To find out more,<br /><ol><br /><li>Read about <a href="http://www.terracotta.org/">Terracotta</a></li><br /><li>Check out some bite-sized code samples in the <a href="http://www.terracotta.org/confluence/display/howto/Cookbook">cookbook section</a></li><br /><li>Or just <a href="http://www.terracotta.org/confluence/display/orgsite/Download">download it already</a> :)</li><br /></ol><br /><br />Note that the code posted in this demo is 100% runnable - just<br /><ol><br /><li>save it to Main.java and tc-config.xml in a new directory</li><br /><li>type "javac Main.java",</li><br /><li>start the Terracotta server - start-tc-server.sh&</li><br /><li>run the program - dso-java.sh Main</li><br /></ol></div>Taylorhttp://www.blogger.com/profile/07193759050963768511noreply@blogger.com0