Wednesday, June 17, 2009

How To Optimize Performance (or how to do Performance Testing right)

Optimizing performance requires you to performance test.

I'm just going to say it - performance testing is hard. Really hard.

Ask anyone that's done it before, and they will agree. If you haven't done it before, well, yeah, sorry. It's not easy. But you've got to do it anyway - because the most important thing you will do as a software engineer is performance test. It's a bit like when your Dad told you "when you grow up to be my age <insert age old wisdom here>" and you didn't believe him?

And now you're old enough that you realize, hey, the guy might have had a point?

Yeah, trust me. Performance testing is both the hardest, and most important, thing you will ever do in your software engineering career. Get it right - you'll be a rockstar. Don't do it - well, I promise you, you'll always be griping about why the amazing software you write is never actually used for "production" apps.

So here you go, simple steps to performance engineering:

1) Set goals - what are you trying to accomplish
2) Measure a baseline
3) Identify a bottleneck
4) Fix said bottleneck
5) Repeat until you meet your performance goals

Did I miss anything? Ahhh...yes. TAKE NOTES.

Let's try again:


1) Set goals - what are you trying to accomplish
1a) Take notes
2) Measure a baseline
2a) Take notes
3) Identify a bottleneck
3a) Take notes
4) Fix said bottleneck
4a) Take notes
5) Repeat until you meet your performance goals

Step 6) -- Report to your boss how much better your application is. But because of Step 1, you'll be able to tell him/her why it matters, right? :)

Friday, April 24, 2009

Terracotta and Spring - Powering High Throughput JEE Applications

Recently, Terracotta did a webinar with Spring founder Colin Sampaleanu.

The webinar starts out by covering the benefits that a Spring+Hibernate+Terracotta application can deliver for your Java JEE application. The latter half is dedicated to running through a reference application that provides a solid starting point as you explore all that Terracotta+Spring can provide.

Examinator

The application demonstrated in the webinar is called "Examinator", and was jointly developed by SpringSource and Terracotta. Briefly,

Examinator is a full stack reference application which demonstrates with code how to build a higly scalable, highly available application using Spring, Hibernate and Terracotta


Highlights include:
  • Frameworks: Spring MVC, Spring Security, Spring Web Flow, Hibernate

  • Scale: 16 application servers, 20,000 concurrent users

  • Latency: Max of 5 ms response time

Find out more

For a full-length recording of the Webinar available for free visit http://terracottech.webex.com.

For a complete reference—everything you need to know including full documentation, how install and run Examinator— visit http://www.terracotta.org.

You can also access a live demo of Examinator at reference.terracotta.org.

SpringOne 2009

Speaking of Spring, I'll be attending the Terracotta booth at SpringOne Europe 2009 this coming week (April 27-29, 2009) in Amsterdam. Stop by if you're attending.

Wednesday, April 22, 2009

A simple tip for new Terracotta users - always run the Terracotta Developer Console

With the release of Terracotta 3.0, I hope many of you have or are considering checking out Terracotta to see if it can help with scalability and availability of your Java application.

Of course www.terracotta.org - in particular the tutorials section with many simple recipes for exploring the many uses of Terracotta is a good place to get started.

But before you do any of that, I'd like to point out a best practice for learning and working with Terracotta. So, here's my tip for whenever you are working with Terracotta:

TIP: ALWAYS RUN THE TERRACOTTA DEVELOPER CONSOLE

It's easy to do, so I recommend before you run any samples, try an recipes, or work with your application, make sure to have the Developer Console running at all times.

How to run the Terracotta Developer Console


Running the Developer Console is easy. There are many ways depending on your context:

  • From the Welcome Application: Click the "Developer Console" link

  • From the command line: Run $TC_HOME/bin/dev-console.sh|bat

  • From Maven: Run $ mvn tc:dev-console

  • From Eclipse: Select Show Developer Console from the Terracotta menu.

Once you've got the Developer Console running, make sure to select the Connect automatically checkbox before connecting—this option will automatically connect the Developer Console to your cluster meaning you don't have to select "connect" every time you run a new cluster instance. This is very useful during experimentation (running sample demos and recipes) and integration testing.

Why should you run the Terracotta Developer Console?

We designed it so that you have access to a large array of information right at your fingertips. In particular, let's look at the user interface which is new in Terracotta 3.0:



One thing that I hope jumps out at you immediately is the presence of a new array of "speedo" dials - somewhat like the array of instruments that greets you when you step into the driver's seat of an automobile.

The resemblance isn't accidental. Those dials are there to give you up-to-the-second information about what's going on in the cluster - and to help pinpoint a problem - if there is one. Let's take a closer look:

Making use of the Speed Dials




As you can see, the dials are arrayed from left to right, giving you vital statistics about the cluster. The dials are separated into two groups:

  • Write Transacions

  • Impeding Factors

The Write Transactions dial contains a measure of the number of write transactions that are occurring in the system. Read transactions with Terracotta are exceedingly cheap (so cheap in fact that we don't measure them). Write transactions are a good measure of work being done in the cluster - so this measure is effectively a measure of how fast your application is running.

The Impeding Factors set of dials shows you a set of seven dials that show you statistics about other types of activity in the system. The activities displayed include such statistics as Object Creation Rate/s — the amount of new objects being added to the clustered heap per second — and Lock Recalls/s — the amount of lock requests that are being transferred from one client node to another.

Making use of the Runtime Statistics

Another very useful feature is the Runtime statistics panel. You can access this feature from the left menu tree by selecting the Runtime Statistics node.



The runtime statistics give you access to a wealth of realtime data with historical views. Unlike the Speedo Dials, the runtime statistics are kept for a longer period of time and graphed for you so you can see a historical view of how your application has been behaving.

Putting it all together

The Speed Dials give you instantaneous information - so they are visible all the time.

Look at the Write Transactions to measure your speed, and monitor the Impeding Factors to make sure nothing is slowing you down.

If there's something worth looking at in more detail, switch to the runtime statistics for more detailed information.

If there is a problem worth investigating, use the Diagnostics tools such as the Lock Profiler or Cluster Wide Thread Dump to debug further.

In other words ALWAYS RUN THE TERRACOTTA DEVELOPER CONSOLE!

Sunday, December 14, 2008

Simple Java Messaging

Following up on my recent post Java Distributed Lock Manager, sometimes you just need a simple way to pass messages between Java processes.

Messaging is a very useful pattern in Enterprise Integration, and there are many ways to do it. Apache Camel is a great tool when you need the flexibility and power to manage complex messaging patterns, including routing, filtering and the like.

If you just want to do something simple, though, that can be a challenge. The most common solution, JMS, requires quite a bit of boilerplate code, and requires selecting and running a JMS provider, which means selecting a J2EE container, Apache ActiveMQ, or others.

So what if you just want a drop-dead simple way of adding messaging to your application? Terracotta gives you that. (And also integrates well with other solutions, like Apache Camel if you need more power later on).

Simple messaging in Terracotta is built on the notion of clustering a LinkedBlockingQueue. Just as a LinkedBlockingQueue is used to pass messages between threads in a single JVM, it will be used in combination with Terracotta's JVM-level clustering to provide message passing between JVMs.

To demonstrate, here is a simple example.

import java.io.*;
import java.util.concurrent.*;
import java.util.concurrent.locks.*;

public class SimpleMessage
{
private static ReentrantReadWriteLock lock = new ReentrantReadWriteLock();
private static BlockingQueue<String> queue = new LinkedBlockingQueue<String>();

public static void receive() throws InterruptedException
{
System.out.println("Receiving messages...");
while (true) {
String msg = queue.take();
System.out.println("msg >> " + msg);
}
}

public static void send() throws Exception
{
while (true) {
System.out.print("Enter a message> "); System.out.flush();
String msg = new BufferedReader(new InputStreamReader(System.in)).readLine();
queue.put(msg);
}
}

public static void main(String[] args) throws Exception
{
// we use the presence of a lock to distinguish receiver from sender
if (lock.writeLock().tryLock()) {
receive();
} else {
send();
}
}
}
The app consist of two modes - a receiver mode and a sender mode. Normally, you would have an application specific mechanism of choosing whether you wanted to send messages or receive messages. For this example, we use a simple lock (for more information on using a ReentrantReadWriteLock with Terracotta, read the ReentrantReadWriteLock recipe). When free, the lock indicates no processes are receiving messages, so the process takes on the "receiver" mode. All subsequent processes take on the "sender" mode when the lock is held.

So let's run it with Terracotta and see how it works. First, we need to "cluster" the app. We need the lock and queue objects to be the same cluster-wide, which in Terracotta is called a root. So our Terracotta configuration file looks like:
<tc:tc-config xmlns:tc="http://www.terracotta.org/config"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.terracotta.org/schema/terracotta-4.xsd">

<application>
<dso>
<roots>
<root>
<field-name>SimpleMessage.lock</field-name>
</root>
<root>
<field-name>SimpleMessage.queue</field-name>
</root>
</roots>
</dso>
</application>
</tc:tc-config>
Now, let's run two JVMs with Terracotta. First, we start a server instance:
$ start-tc-server.sh
2008-12-14 10:26:18,246 INFO - Terracotta Server has started up as ACTIVE node on
0.0.0.0:9510 successfully, and is now ready for work.
Then, we start our JVMs.

JVM 1:
$ dso-java.sh SimpleMessage
Receiving Messages...
JVM 2:
$ dso-java.sh SimpleMessage
Enter a message>
Here, we enter a message, and see that it is printed in JVM 1:

JVM 2:
$ dso-java.sh SimpleMessage
Enter a message> hello world
JVM 1:
$ dso-java.sh SimpleMessage
Receiving Messages...
msg >> hello world
Further exploration

Try starting another JVM and see that they can both send messages to JVM 1. Try killing the receiver JVM and send messages to it. Then start another JVM. Since the lock is no longer held (Terracotta automatically releases any locks held by a JVM that exits the cluster) the new JVM will take on the receiver mode. Any messages sent while there was no receiver will have been queued, and will be printed on the startup of this new node.

And of course, you can see all the activity in the cluster. Try taking the receiver down again, send some messages using the sender nodes, then run the admin console. You'll be able to inspect the messages in the queue using the clustered heap browser.

This is just a demonstration of course - so to keep it simple I used a String as the message - but you could use any class.

For more fun with Terracotta, try the helpful "recipes" at Terracotta.org.

(Note, I've blogged about simple coordination in the past using Terracotta, which is similar)

Thursday, December 11, 2008

Java Distributed Lock Manager

Sometimes you just need a simple way to coordinate activities across more than one java process. There's a lot of choices out there. The database, JMX, distributed caches, JMS, filesystems. It would be nice if there was a simple, easy way to get distributed locks in a J2SE, J2EE, Web, SOAP, or AJAX application? There is.

Terracotta provides one of the easiest ways to get a distributed lock manager in your Java application. Terracotta plugs right in to normal Java threading constructs—synchronized, wait/notify, java.concurrent.locks.ReentrantReadWriteLock, and even java.concurrent.CyclicBarrier, which means you basically already know how to use Terracotta as a lock manager.

To demonstrate, let's work up a simple locking example and then drop Terracotta in. Our app will consist of acquiring a lock, "do some work" in a simple loop, and repeat. Here's the code (LockExample.java):

public class LockExample
{
private static Object lock = new Object();

public static void main(String[] args) throws Exception
{
while (true) {
System.out.print("Waiting for the lock..."); System.out.flush();
synchronized (lock) {
System.out.print("I got the lock, doing work");
for (int i = 0; i < 4; i++) {
Thread.currentThread().sleep(1000);
System.out.print("."); System.out.flush();
}
}
System.out.println("done");
}
}
}
Simple. If we run this on the command line, we get:
$ javac LockExample.java
$ java LockExample
Waiting for the lock...I got the lock, doing work....done
Waiting for the lock...I got the lock, doing work....done
During the "work" part the "."'s are added 1 every second for four seconds. Fancy.

Let's add Terracotta. We need a tc-config.xml file which tells Terracotta how to provide the appropriate clustering behavior to our application:
<tc:tc-config xmlns:tc="http://www.terracotta.org/config"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.terracotta.org/schema/terracotta-4.xsd">

<application>
<dso>
<locks>
<autolock>
<method-expression>void LockExample.main(..)</method-expression>
</autolock>
</locks>
<roots>
<root>
<field-name>LockExample.lock</field-name>
</root>
</roots>
</dso>
</application>
</tc:tc-config>
Now, let's run two JVMs with Terracotta. First, we start a server instance:
$ start-tc-server.sh
2008-12-11 22:26:18,246 INFO - Terracotta Server has started up as ACTIVE node on
0.0.0.0:9510 successfully, and is now ready for work.
Then, we start our JVMs.

JVM 1:
$ dso-java.sh LockExample
Waiting for the lock...I got the lock, doing work....done
JVM 2:
$ dso-java.sh LockExample
Waiting for the lock...
It's a bit hard to demonstrate in a blog post, but the lock ping-pongs between the JVMs. That's it!

For more fun with distributed lock coordination, try these helpful "recipes":

Saturday, November 22, 2008

Sales 101: The 5 sales archetypes

There's a lot they don't teach you in school. As an engineer, most of the things I rely on day to day they never even so much as mentioned when I was in college, things like bug tracking, revision control, heck even writing error messages.

If you're an engineer, and in sales, it's even worse. I have spent the better half of my career in pre and post-sales as a "solutions architect" before moving on to product management. One thing that I have found to be useful is how to identify the person you are talking to on the other end of the phone.

I am not talking about figuring out if you are talking to a developer, a manager, an architect or a CEO. I mean what kind of person is this - what problems do they have, what kind of ego do they have, but most importantly, are they actually going to spend money. Like it or not, these considerations make a major contrbution to chances of your success.

So here's the 5 kinds of people you are most likely to meet, how to identify them, and what they mean to the bottom-line.

1. The beard-tugger

Summary: The beard-tugger thinks he is smarter than everyone else, and is committed to proving it. Thus he will spend the entire sales-pitch showing you just how smart he is.

Tell tale signs you have a beard-tugger: For every feature you talk about in your product, the beard-tugger will analyze it in 5 different ways and 10 different contexts. If he sees the slightest hole in your theory or implementation, he's bound to ask a question about it.

Pros: If you like to get into the guts of the product with someone, this is a great person to do it with. Remember though that if you are trying to sell this person, you need to let them "win" - e.g. never show them up to be lesser than the audience, or you risk losing them as a champion of your product.

Cons: There is no faster way to sink a pitch than to rat-hole on some minor detail of your product. The beard-tuggers middle name is rat-hole, so be careful.

2. The tire-kicker

Summary: The tire-kicker is out for a good time. They will take a sales call from anyone, and don't really have an agenda. They just want to see what you have. Often, this person thinks of themselves as "knowledgable about the market" so they will talk to you just to get a feel for your product to reinforce that feeling.

Tell tale signs you have a tire-kicker: if your product can be used in 10 different ways, he wants to know about every one of them. If you ask them about the specific problem they are looking to solve (and you should) the answer will be vague or none at all.

Pros: none really. This person may ultimately be a champion for you when they come around to solving a problem, so don't shun them. But you should get on with your life as soon as you can. Pitch your wares succinctly and move on.

Cons: big time suck. If you are not careful this person could come back time and time again, without a real problem to solve. That could be a big time sink for no real opportunity for business, in other words, a real waste of your time.

3. The science-experiment

Summary: Similar to the tire-kicker, but a little more involved, the science-experimenter is likely to have a problem in mind and wants to solve it, but there's generally no business behind it. This person is likely to be "exploring" technologies, but has picked yours as a likely candidate for a solution.

Tell tale signs you have a science-experiment: this person is allocating resources to a project to test out your product - e.g. run a POC (Proof of Concept). But if you press them, you will find there are no hard and fast requirements, so everything is either made up or guesswork. This person is likely to be enthusiastic both about their use case, and your product. They are probably even more enthusiastic about the opportunity for the mutual relationship to grow.

Pros: If you are successful, this could blossom into a sale, and the science-experimenter will probably tell you that at least 5 times in the span of 20 minutes.

Cons: A science-experiment goes nowhere 9 times out of 10. Stay away because there are no real requirements, and even if you succeed (which is unlikely, given the lack of a real requirement or business driver) there is unlikely to be a pot of gold at the end of this relationship, despite the claims of the science-experimenter to the contrary.

4. The delusional

Summary: This person is trying to use your product either for an outlandish use case, in an extreme way, or worse, in every possible way (e.g. they think you have the silver-bullet to every problem known to man).

Tell tale signs you have a person with delusions of grandeur on the line is that they will be obsessed with how successful their product will be. That is to say, they have either no product, or a very small product at the moment, and this person is most likely obsessed with the massive growth they are just about to incur.

Pros: None. This is one of the most dangerous people to engage with. Get out fast.

Cons: Very likely to be enthusiastic with your product and want to POC it. This is the most dangerous of people, because they will be pandering to your ego, which means you will be very easily swayed by them because they believe in you, and they love your product. They will spend a very long time in a POC with your product, because the requirements will likely be unrealistic bordering on ridiculous. If you bend to their whims, you will likely be adding features/fixing bugs that have no bearing on real business.

5. Your customer

Summary: This is the person you want to sell to. They have a business case, and likely some pain. When you discover what that pain is, you should have a product that solves that pain, for less money then they are currently spending.

Tell tale signs you are talking to your ideal customer: at the end of your initial call with this person, they should have asked you how much your product costs, and they should be wary that your product can actually solve their problem (afterall, nothing else has to date), but be slightly optimistic that maybe you can solve their problem, and be willing to try. They should have a use case that is within the bounds of what your product can solve, and they should be interested in what the next steps are after the call.

Pros: Sell them quickly - this is where you should be spending your time.

Cons: None - your only challenge is to identify this person. If you don't, you're losing money.

Now, go out there and make some money! :)

Monday, November 17, 2008

Grails + Quartz + Terracotta

1) Grails recently added plug-in support for Terracotta.

2) Grails recently added plug-in support for Quartz.

3) Terracotta supports Quartz

So....wouldn't it be possible to demonstrate Grails, Quartz and Terracotta all working together? Seems like a fun project.