New Features in ActiveMQ 5.4: Automatic Cluster Update and Rebalance

Apache ActiveMQ 5.4.0 was released in August, followed quickly in September by a 5.4.1 release. Not only were there tons of fixes in these releases, but there were also some really great new features including message scheduling, support for web sockets, new Unix control scripts, full message priority, producer message caching and cluster client updates and cluster client rebalancing just to name a few. In this blog post, I'm going to discuss the new cluster client updates and cluster client rebalancing features so that you get a taste of how they are used.

Problem Introduction

When using a network of brokers with ActiveMQ, the configuration of the brokers that form the network has always been rather static. The first step toward a more dynamic network of brokers was a feature I presented in a previous blog post titled How to Use Automatic Failover In an ActiveMQ Network of Brokers. In the event of a broker connection failure, the use of the failover transport for the network connections between brokers allows those connections to be automatically reestablished. This is a wonderful feature for sure, but it only got us part of the way toward a truly dynamic cluster of brokers.

Two new features in ActiveMQ 5.4 introduce the concept of making the cluster of brokers even more dynamic. These two items are the ability to:

• Update the clients in the cluster
• Rebalance the brokers in the cluster

Both of these features are quite interesting so I will explain how each one works.

Update Cluster Clients

In the past, when clients connected to brokers in the cluster, it was recommended to keep a comma-separated list of broker URIs in the failover transport configuration. Below is an example of this style of configuration:

failover:(tcp://machineA:61616,tcp://machineB:61616,tcp://machineC:61616)?randomize=false

The failover configuration example above lives on the client side and contains a static list of the URIs for each broker in the cluster. In the event that a broker in the cluster fails, the failover transport is what allows a client to automatically reconnect to another broker in that list of broker URIs. Unfortunately this style of configuration can be difficult to maintain because it is static. If you want to add another broker to the cluster, every client's failover transport configuration must be updated manually. Depending on the number of clients in your cluster, this could really be a maintenance headache. This is where the first new features comes to the rescue.

ActiveMQ 5.4 provides the ability to automatically update the clients in the cluster. That is, if a new broker joins or leaves the existing network of brokers, the clients' failover transport configurations no longer need to be manipulated manually. Using configuration options on the broker, you can tell the broker to update each client's failover transport configuration automatically. Below is an example of this new feature:


<broker brokerName="brokerA" ...>
...
 <transportConnectors>
   <transportConnector name="tcp-connector" uri="tcp://192.168.0.23:61616" updateClusterClients="true" />
 </<transportConnectors>
...
</broker>


The configuration above is on the broker-side. Notice the new attribute in the <transportConnector> element named updateClusterClients=true. This attribute is used in conjunction with the failover transport on the client-side and it tells the broker to automatically update the client's failover transport configuration when the network topology changes. In addition to the updateClusterClients=true property, there are also a few others including:

• updateClusterClientsOnRemove - Updates a client when brokers are removed from the cluster.
• updateClusterFilter - A comma-separated list of regexes to match broker names that are part of the cluster. This allows flexibility for the inclusion/exclusion of brokers.
• updateURIsURL - Used to provide the path to a file containing a comma-separted list of broker URIs.

These new features are extremely powerful because they allow for a much more dynamic network of brokers configuration. Anyone who has had to deal with the static nature of the failover transport configuration should understand the power in these new features and do some experimentation to see how they operate.

Rebalance Cluster Clients

The second new feature also builds upon the failover transport configuration, but for a slightly different purpose. Consider the fact that when a new broker is added to/removed from the cluster that clients cannot automatically take advantage of it. Even with the new ability to update the clients so that they have knowledge of the broker being added/removed, there was no way previously for them to actually use that broker unless a failure occurred. Well that's what this feature does.

ActiveMQ 5.4 allows clients to be automatically disconnected from their current broker and reconnect to a different broker. Here's an example to illustrate this feature. Let's say you have a cluster of three brokers: brokerA, brokerB and brokerC, each of which has some clients connected. When a new broker is added to the cluster, if the updateClusterClients property is set to true, then the clients will be notified about the new broker, but no action will be taken unless the rebalanceClusterClients property is set to true. When the rebalanceClusterClients property is set to true, the clients will be automatically be disconnected from their current broker in order to reconnect to another broker in the cluster. Below is an example configuration for the new rebalance property:


<broker brokerName="brokerA" ...>
...
 <transportConnectors>
   <transportConnector name="tcp-connector" uri="tcp://192.168.0.23:61616" updateClusterClients="true" rebalanceClusterClients="true" />
 </<transportConnectors>
...
</broker>


Notice the new rebalanceClusterClients attribute in the <transportConnector> element. This property enables the clients to immediately take advantage of the new broker in the cluster. Instead of waiting for the next connection failure and a reconnect from the failover transport, the clients are told to reconnect immediately to another broker in their list.

Testing The New Features

Testing these two new features is pretty easy actually. Below are the steps I have used on a few occasions:

1. Make sure that your clients are logging the broker URI to which they are connected for sending or receiving messages
2. Configure each client to only have one broker URI in its failover transport configuration
3. Configure the transport connector on the broker-side to set the updateClusterClients property to true and the rebalanceClusterClients property to true
4. Start up the brokers in your cluster
5. Start up the clients that connect to a broker in the cluster
6. Add a new broker to the cluster and observe the following behavior:

Due to the two new properties that have been set on the broker-side, each client will be notified of the new broker that was added to the cluster AND each client will automatically reconnect. That is, the functionality of the failover transport will be engaged so that each client is disconnected from the current broker and reconnected to another broker in the list (i.e., the list of broker URIs in the failover transport configuration).

The fact that each client reconnects to a new broker tells you that:

1. The updateClusterClients property is working correctly because you should see the logging change from one broker URI to another. Remember that each client was started with only one broker URI in their failover transport config. The fact that they are reconnecting tells you that they are receiving notifications of changes to the cluster.
2. The rebalanceClusterClients property is working properly because the clients reconnected.

Verify this using the logging from each client. You will see that each client was sending or receiving messages to/from one broker URI and suddenly the logging changes to another broker URI. This tells you that the clients are being updated and rebalanced.

Conclusion

These new features are quite powerful additions to the ActiveMQ network of brokers. They really advance ActiveMQ beyond the static configurations upon which we have all relied for many years now. Most likely the sys admins and dev ops folks will enjoy these features the most because they will no longer need to manually manage a static list of broker URIs for clients.

As I said earlier, many other great features were also introduced in ActiveMQ 5.4 and 5.4.1. So try them out yourself to see if they help to improve your application development.

Update: If you only set the updateClusterClients="true" and the rebalanceClusterClients="true" options, you will notice that when a broker in the network fails and is brought back up, the client connections to other brokers in the network are not automatically rebalanced. This is due to the lack of the updateClusterClientsOnRemove="true" option. After adding this option to the config, network broker clients are notified of broker failures which basically completes the circle and allows the automatic rebalancing to work as it should.

40% Off ActiveMQ In Action To Celebrate Going Into Production!

Good news -- ActiveMQ In Action has now entered the Manning production process! This means that we are done with the writing of content for the book and it is currently being copy edited and undergoing technical review. In fact, we are currently seeking reviewers for the book. If you are interested to review the book, please send an email to mkt@manning.com.

Also, we are currently offering 40% off your purchase of ActiveMQ In Action! Just use the coupon code *activemq40* at the time of checkout. Hurry and make your purchase today as this discount won't last long!

SpringOne 2GX 2010 in Chicago

Last week I attended our annual SpringOne 2GX developer conference and I saw many excellent presentations by many different folks. As is common at a developer conference, there were numerous times where I was torn by the decision to attend one talk and not another due to time scheduling. Probably the most intriguing talk I attended was a Technical deep-dive of hypervisors and virtualization for developers by Richard McDougall. I learned a lot about virtualization that I didn't know and, as always, Richard's depth of knowledge on this topic is completely impressive! But my favorite talk was one delivered by my friends Dave Syer and Mark Fisher titled Concurrent and Distributed Applications with Spring.

I always enjoy discussions, books, articles, etc. on the topic of Java concurrency, but Dave and Mark's session was different. In the same vein as the Java Concurrency in Practice book, Dave and Mark brought a practicality to the topic that really scored a goal with the attendees. When I go to conferences I tend to judge sessions not solely based on my own opinion, but moreso on the opinion of other attendees. Not only was this session completely packed, but it stayed packed the entire time, i.e., people were not walking out early. This was due to the way that Dave and Mark delivered a difficult topic in a way that all Java developers can understand. What many of the conference sessions, books, articles, etc. have in common when presenting Java concurrency is that they don't seek to simplify the content and present it in a way that makes it easier to understand. Dave and Mark really tore down most of the complexity with Java concurrency and presented it in a way that makes it seem approachable. In fact, I saw more people taking notes in this session than in any other that I attended. They also provided hands-on examples that were simplified and to the point. I think that they could have gone on discussing this topic for another 90 minutes and people would have stayed. Now, aren't you jealous that you weren't there?!

Other excellent sessions included:

• Gaining visibility into enterprise Spring applications with tc server Spring Edition by Steve Mayzak
• Clustering and load-balancing with tc Server and httpd by Mark Thomas
• Mastering MVC 3 by Keith Donald
• Developer Tools to push your Productivity by Andy Clement and Christian Dupuis
• What's new in Spring Integration 2.0? by Mark Fisher and Oleg Zhurakousky
• Diagnosing Performance Issues, with Spring Insight, Before it's a Problem by Scott Andrews and Jon Travis
• Spring and Java EE 6 by Juergen Hoeller
• Payments in one API by John Davies and Rossen Stoyanchev
• How to build business applications using Google Web Toolkit and Spring Roo by Amit Manjhi
• Harnessing the power of HTML5 by Scott Andrews and Jeremy Grelle
• Case Study: EMC Next-Generation ERP Integration Architecture by Brian Dussault and Tom McCuch
• Case Study: Migrating Hyperic HQ from EJB to Spring by Jennifer Hickey
• Monitoring Spring Batch and Spring Integration with SpringSource Hyperic by Dave Syer

There were also tons of really great Groovy and Grails talks such as:

• Transforming to Groovy by Venkat Subramaniam
• GORM Inside And Out by Jeff Brown
• Improving your Groovy Code Quality by Venkat Subramaniam
• Tuning Grails applications by Peter Ledbrook
• Unit and Functional Testing using Groovy by Venkat Subramaniam
• Groovy and Concurrency by Paul King
• Tomorrow's Tech Today: HTML 5 + Grails by Scott Davis

So if you didn't make it to SpringOne this year, now you have just a taste of what you missed. And hopefully you have good reason to consider attending next year. Any guesses on where the conference might take place next year? Of course, I'm rooting for Boulder, CO. Adam? Are you listening? ;-)