Thursday, October 01, 2009

A simple load test in Terracotta...

This is a response to the following blog in which the author wrote a micro-benchmark and got some pretty bad results using Terracotta http://zion-city.blogspot.com/2009/10/terracotta-as-distributed-dbms-bad-idea.html.

Since the commenting system on blogger doesn't allow code, I am posting the response on my blog with code attached for reference.

So my approach was to try replicate the author's implementation, to see what kind of performance a straightforward micro-benchmark might achieve.

Reader beware - micro-benchmarks are never a good idea, and not usually indicative of real-world performance. In this case, based on real-world results I have seen, my results appear to be a lower bound for the kind of performance one should expect since the test isn't concurrent and is running on a single machine - hardly the kind of environment a real world clustered app would exist in)

So, with that said, I wrote a simple load test against a ConcurrentHashMap, and put 100,000 objects into it.

My results show:
Avg TPS: ~3,000
Instantaneous TPS as high as: ~7,000

Here's the code:

import java.util.Date;
import java.util.Map;
import java.util.concurrent.*;

public class Main
{
static Map<Integer, Foo> map =
new ConcurrentHashMap<Integer, Foo>();

public static class Foo
{
public String name;
public String name2;
public String name3;

public Foo(String name)
{
this.name = name;
this.name2 = name + " 2";
this.name3 = name + " 3";
}
}

public static void main(String[] args)
{
long start = System.currentTimeMillis();

for (int i = 0; i < 100000; i++) {
map.put(i, new Foo(new Date().toString()));
}
System.out.println("elapsed: " + (System.currentTimeMillis() - start));
}
}


And here's the tc-config.xml:
<?xml version="1.0" encoding="UTF-8"?>
<tc:tc-config xmlns:tc="http://www.terracotta.org/config"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.terracotta.org/schema/terracotta-5.xsd">
<application>
<dso>
<instrumented-classes>
<include>
<class-expression>Main$Foo</class-expression>
</include>
</instrumented-classes>
<roots>
<root>
<field-name>Main.map</field-name>
</root>
</roots>
</dso>
</application>
</tc:tc-config>


I took a screenshot of the dev console running during the test, to give you an idea of the instantaneous TPS achieved:

Wednesday, September 02, 2009

Great customer service in the cloud

It's interesting to see providers moving to and, by proxy, the rest of us relying on, the cloud.

I just spent a few hours at VMWorld, and judging by the size, sophistication, and variety of providers, vendors, products, and companies, virtualization technologies, and in particular, cloud computing, is here to stay.

Today, Google apologized for it's GMail outage yesterday, with a completely forthright, mature, and encouraging response.


Gmail's web interface had a widespread outage earlier today, lasting about 100 minutes. We know how many people rely on Gmail for personal and professional communications, and we take it very seriously when there's a problem with the service. Thus, right up front, I'd like to apologize to all of you — today's outage was a Big Deal, and we're treating it as such. <read the rest of Google's post...>


And, in a twist of fate, just a few days ago I received an email from Netflix. Apparently they had some trouble with their network while I was trying to watch a tv show using my XBox 360. Not only did they figure this out, but they sent me an email that offered to discount my monthly fee by 3%. This is fantastic customer service! Here's the email in it's entirety:


We're sorry you may have had trouble watching instantly via your Xbox

Dear Taylor,

Last night, you may have had trouble instantly watching movies or TV episodes via your Xbox due to technical issues.

We are sorry for the inconvenience this may have caused. If you were unable to instantly watch a movie or TV episode last night via your Xbox, click on this account specific link in the next 7 days to apply your 3% credit on your next billing statement. Credit can only be applied once.

Again, we apologize for any inconvenience, and thank you for your understanding. If you need further assistance, please call us at 1-866-923-0898.

-The Netflix Team


Failures do happen. And today's scaled-out architectures are designed to be resilient to these failures. But the fact is that even though these designs exist, and are generally very resilient against failures, giving these services availability times numbered in the 9's, mistakes in design, implementation, or execution still do happen.

I say, give these guys massive cred for owning up to their mistakes, and dealing with the consumer in an open and honest way. That's the way to build solid relationships, and I for one will not look twice when Yahoo! or Blockbuster sends me that next request to join up on their service. These guys have it figured out, and have sold me as lifelong* customer, even if they're not perfect.

If only legacy infrastructure (power, cable (grrr Comcast), telephone (AT&T - I'm looking at you)) and the like could understand the value in this approach.

* lifelong in tech years, which is only about 5 years or so ;-)

Wednesday, June 17, 2009

How To Optimize Performance (or how to do Performance Testing right)

Optimizing performance requires you to performance test.

I'm just going to say it - performance testing is hard. Really hard.

Ask anyone that's done it before, and they will agree. If you haven't done it before, well, yeah, sorry. It's not easy. But you've got to do it anyway - because the most important thing you will do as a software engineer is performance test. It's a bit like when your Dad told you "when you grow up to be my age <insert age old wisdom here>" and you didn't believe him?

And now you're old enough that you realize, hey, the guy might have had a point?

Yeah, trust me. Performance testing is both the hardest, and most important, thing you will ever do in your software engineering career. Get it right - you'll be a rockstar. Don't do it - well, I promise you, you'll always be griping about why the amazing software you write is never actually used for "production" apps.

So here you go, simple steps to performance engineering:

1) Set goals - what are you trying to accomplish
2) Measure a baseline
3) Identify a bottleneck
4) Fix said bottleneck
5) Repeat until you meet your performance goals

Did I miss anything? Ahhh...yes. TAKE NOTES.

Let's try again:


1) Set goals - what are you trying to accomplish
1a) Take notes
2) Measure a baseline
2a) Take notes
3) Identify a bottleneck
3a) Take notes
4) Fix said bottleneck
4a) Take notes
5) Repeat until you meet your performance goals

Step 6) -- Report to your boss how much better your application is. But because of Step 1, you'll be able to tell him/her why it matters, right? :)

Friday, April 24, 2009

Terracotta and Spring - Powering High Throughput JEE Applications

Recently, Terracotta did a webinar with Spring founder Colin Sampaleanu.

The webinar starts out by covering the benefits that a Spring+Hibernate+Terracotta application can deliver for your Java JEE application. The latter half is dedicated to running through a reference application that provides a solid starting point as you explore all that Terracotta+Spring can provide.

Examinator

The application demonstrated in the webinar is called "Examinator", and was jointly developed by SpringSource and Terracotta. Briefly,

Examinator is a full stack reference application which demonstrates with code how to build a higly scalable, highly available application using Spring, Hibernate and Terracotta


Highlights include:
  • Frameworks: Spring MVC, Spring Security, Spring Web Flow, Hibernate

  • Scale: 16 application servers, 20,000 concurrent users

  • Latency: Max of 5 ms response time

Find out more

For a full-length recording of the Webinar available for free visit http://terracottech.webex.com.

For a complete reference—everything you need to know including full documentation, how install and run Examinator— visit http://www.terracotta.org.

You can also access a live demo of Examinator at reference.terracotta.org.

SpringOne 2009

Speaking of Spring, I'll be attending the Terracotta booth at SpringOne Europe 2009 this coming week (April 27-29, 2009) in Amsterdam. Stop by if you're attending.

Wednesday, April 22, 2009

A simple tip for new Terracotta users - always run the Terracotta Developer Console

With the release of Terracotta 3.0, I hope many of you have or are considering checking out Terracotta to see if it can help with scalability and availability of your Java application.

Of course www.terracotta.org - in particular the tutorials section with many simple recipes for exploring the many uses of Terracotta is a good place to get started.

But before you do any of that, I'd like to point out a best practice for learning and working with Terracotta. So, here's my tip for whenever you are working with Terracotta:

TIP: ALWAYS RUN THE TERRACOTTA DEVELOPER CONSOLE

It's easy to do, so I recommend before you run any samples, try an recipes, or work with your application, make sure to have the Developer Console running at all times.

How to run the Terracotta Developer Console


Running the Developer Console is easy. There are many ways depending on your context:

  • From the Welcome Application: Click the "Developer Console" link

  • From the command line: Run $TC_HOME/bin/dev-console.sh|bat

  • From Maven: Run $ mvn tc:dev-console

  • From Eclipse: Select Show Developer Console from the Terracotta menu.

Once you've got the Developer Console running, make sure to select the Connect automatically checkbox before connecting—this option will automatically connect the Developer Console to your cluster meaning you don't have to select "connect" every time you run a new cluster instance. This is very useful during experimentation (running sample demos and recipes) and integration testing.

Why should you run the Terracotta Developer Console?

We designed it so that you have access to a large array of information right at your fingertips. In particular, let's look at the user interface which is new in Terracotta 3.0:



One thing that I hope jumps out at you immediately is the presence of a new array of "speedo" dials - somewhat like the array of instruments that greets you when you step into the driver's seat of an automobile.

The resemblance isn't accidental. Those dials are there to give you up-to-the-second information about what's going on in the cluster - and to help pinpoint a problem - if there is one. Let's take a closer look:

Making use of the Speed Dials




As you can see, the dials are arrayed from left to right, giving you vital statistics about the cluster. The dials are separated into two groups:

  • Write Transacions

  • Impeding Factors

The Write Transactions dial contains a measure of the number of write transactions that are occurring in the system. Read transactions with Terracotta are exceedingly cheap (so cheap in fact that we don't measure them). Write transactions are a good measure of work being done in the cluster - so this measure is effectively a measure of how fast your application is running.

The Impeding Factors set of dials shows you a set of seven dials that show you statistics about other types of activity in the system. The activities displayed include such statistics as Object Creation Rate/s — the amount of new objects being added to the clustered heap per second — and Lock Recalls/s — the amount of lock requests that are being transferred from one client node to another.

Making use of the Runtime Statistics

Another very useful feature is the Runtime statistics panel. You can access this feature from the left menu tree by selecting the Runtime Statistics node.



The runtime statistics give you access to a wealth of realtime data with historical views. Unlike the Speedo Dials, the runtime statistics are kept for a longer period of time and graphed for you so you can see a historical view of how your application has been behaving.

Putting it all together

The Speed Dials give you instantaneous information - so they are visible all the time.

Look at the Write Transactions to measure your speed, and monitor the Impeding Factors to make sure nothing is slowing you down.

If there's something worth looking at in more detail, switch to the runtime statistics for more detailed information.

If there is a problem worth investigating, use the Diagnostics tools such as the Lock Profiler or Cluster Wide Thread Dump to debug further.

In other words ALWAYS RUN THE TERRACOTTA DEVELOPER CONSOLE!

Sunday, December 14, 2008

Simple Java Messaging

Following up on my recent post Java Distributed Lock Manager, sometimes you just need a simple way to pass messages between Java processes.

Messaging is a very useful pattern in Enterprise Integration, and there are many ways to do it. Apache Camel is a great tool when you need the flexibility and power to manage complex messaging patterns, including routing, filtering and the like.

If you just want to do something simple, though, that can be a challenge. The most common solution, JMS, requires quite a bit of boilerplate code, and requires selecting and running a JMS provider, which means selecting a J2EE container, Apache ActiveMQ, or others.

So what if you just want a drop-dead simple way of adding messaging to your application? Terracotta gives you that. (And also integrates well with other solutions, like Apache Camel if you need more power later on).

Simple messaging in Terracotta is built on the notion of clustering a LinkedBlockingQueue. Just as a LinkedBlockingQueue is used to pass messages between threads in a single JVM, it will be used in combination with Terracotta's JVM-level clustering to provide message passing between JVMs.

To demonstrate, here is a simple example.

import java.io.*;
import java.util.concurrent.*;
import java.util.concurrent.locks.*;

public class SimpleMessage
{
private static ReentrantReadWriteLock lock = new ReentrantReadWriteLock();
private static BlockingQueue<String> queue = new LinkedBlockingQueue<String>();

public static void receive() throws InterruptedException
{
System.out.println("Receiving messages...");
while (true) {
String msg = queue.take();
System.out.println("msg >> " + msg);
}
}

public static void send() throws Exception
{
while (true) {
System.out.print("Enter a message> "); System.out.flush();
String msg = new BufferedReader(new InputStreamReader(System.in)).readLine();
queue.put(msg);
}
}

public static void main(String[] args) throws Exception
{
// we use the presence of a lock to distinguish receiver from sender
if (lock.writeLock().tryLock()) {
receive();
} else {
send();
}
}
}
The app consist of two modes - a receiver mode and a sender mode. Normally, you would have an application specific mechanism of choosing whether you wanted to send messages or receive messages. For this example, we use a simple lock (for more information on using a ReentrantReadWriteLock with Terracotta, read the ReentrantReadWriteLock recipe). When free, the lock indicates no processes are receiving messages, so the process takes on the "receiver" mode. All subsequent processes take on the "sender" mode when the lock is held.

So let's run it with Terracotta and see how it works. First, we need to "cluster" the app. We need the lock and queue objects to be the same cluster-wide, which in Terracotta is called a root. So our Terracotta configuration file looks like:
<tc:tc-config xmlns:tc="http://www.terracotta.org/config"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.terracotta.org/schema/terracotta-4.xsd">

<application>
<dso>
<roots>
<root>
<field-name>SimpleMessage.lock</field-name>
</root>
<root>
<field-name>SimpleMessage.queue</field-name>
</root>
</roots>
</dso>
</application>
</tc:tc-config>
Now, let's run two JVMs with Terracotta. First, we start a server instance:
$ start-tc-server.sh
2008-12-14 10:26:18,246 INFO - Terracotta Server has started up as ACTIVE node on
0.0.0.0:9510 successfully, and is now ready for work.
Then, we start our JVMs.

JVM 1:
$ dso-java.sh SimpleMessage
Receiving Messages...
JVM 2:
$ dso-java.sh SimpleMessage
Enter a message>
Here, we enter a message, and see that it is printed in JVM 1:

JVM 2:
$ dso-java.sh SimpleMessage
Enter a message> hello world
JVM 1:
$ dso-java.sh SimpleMessage
Receiving Messages...
msg >> hello world
Further exploration

Try starting another JVM and see that they can both send messages to JVM 1. Try killing the receiver JVM and send messages to it. Then start another JVM. Since the lock is no longer held (Terracotta automatically releases any locks held by a JVM that exits the cluster) the new JVM will take on the receiver mode. Any messages sent while there was no receiver will have been queued, and will be printed on the startup of this new node.

And of course, you can see all the activity in the cluster. Try taking the receiver down again, send some messages using the sender nodes, then run the admin console. You'll be able to inspect the messages in the queue using the clustered heap browser.

This is just a demonstration of course - so to keep it simple I used a String as the message - but you could use any class.

For more fun with Terracotta, try the helpful "recipes" at Terracotta.org.

(Note, I've blogged about simple coordination in the past using Terracotta, which is similar)

Thursday, December 11, 2008

Java Distributed Lock Manager

Sometimes you just need a simple way to coordinate activities across more than one java process. There's a lot of choices out there. The database, JMX, distributed caches, JMS, filesystems. It would be nice if there was a simple, easy way to get distributed locks in a J2SE, J2EE, Web, SOAP, or AJAX application? There is.

Terracotta provides one of the easiest ways to get a distributed lock manager in your Java application. Terracotta plugs right in to normal Java threading constructs—synchronized, wait/notify, java.concurrent.locks.ReentrantReadWriteLock, and even java.concurrent.CyclicBarrier, which means you basically already know how to use Terracotta as a lock manager.

To demonstrate, let's work up a simple locking example and then drop Terracotta in. Our app will consist of acquiring a lock, "do some work" in a simple loop, and repeat. Here's the code (LockExample.java):

public class LockExample
{
private static Object lock = new Object();

public static void main(String[] args) throws Exception
{
while (true) {
System.out.print("Waiting for the lock..."); System.out.flush();
synchronized (lock) {
System.out.print("I got the lock, doing work");
for (int i = 0; i < 4; i++) {
Thread.currentThread().sleep(1000);
System.out.print("."); System.out.flush();
}
}
System.out.println("done");
}
}
}
Simple. If we run this on the command line, we get:
$ javac LockExample.java
$ java LockExample
Waiting for the lock...I got the lock, doing work....done
Waiting for the lock...I got the lock, doing work....done
During the "work" part the "."'s are added 1 every second for four seconds. Fancy.

Let's add Terracotta. We need a tc-config.xml file which tells Terracotta how to provide the appropriate clustering behavior to our application:
<tc:tc-config xmlns:tc="http://www.terracotta.org/config"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.terracotta.org/schema/terracotta-4.xsd">

<application>
<dso>
<locks>
<autolock>
<method-expression>void LockExample.main(..)</method-expression>
</autolock>
</locks>
<roots>
<root>
<field-name>LockExample.lock</field-name>
</root>
</roots>
</dso>
</application>
</tc:tc-config>
Now, let's run two JVMs with Terracotta. First, we start a server instance:
$ start-tc-server.sh
2008-12-11 22:26:18,246 INFO - Terracotta Server has started up as ACTIVE node on
0.0.0.0:9510 successfully, and is now ready for work.
Then, we start our JVMs.

JVM 1:
$ dso-java.sh LockExample
Waiting for the lock...I got the lock, doing work....done
JVM 2:
$ dso-java.sh LockExample
Waiting for the lock...
It's a bit hard to demonstrate in a blog post, but the lock ping-pongs between the JVMs. That's it!

For more fun with distributed lock coordination, try these helpful "recipes":