Why I Love My Bike

Ever since I first learned to ride a bike, I have loved it. My mom and dad taught me to ride a little blue Schwinn bicycle when I was four or five years old. I remember learning very quickly to skid down the sidewalk across some gravel and I thought I was so cool. But just riding my bike made me happy and to this day that is still true.

As a kid, riding a bike meant freedom. I could ride just about everywhere I needed to go in the small town where I grew up in Illinois. And nearly everything we did as kids included riding our bikes. We rode to school, to the park, to soccer practice, to swim practice; I even began going out on the road to neighboring towns. Not only did I enjoy it, but it was good transportation until I was old enough to drive. But even then I kept riding my bike when I could.

During my college years, I bought my first mountain bike. Friends and I would go out riding together in the parks and wooded areas around where we lived and we loved it. Then, a couple years after graduating from college, we moved to Colorado. Then I really got a taste for true mountain biking and it has been in my blood ever since.

When we first moved from Illiniois to Colorado in 1995, one of the reasons we came was for the outdoor activities all year round. Mountain biking was number one on my list and I was out riding in the mountains practically every weekend back then. A few years later, we started our family and our lifestyle really changed (raising kids will do that to you).

When our girls were really young, they had miniature big wheels and then small bikes with training wheels. But once they each got old enough, I got to teach them to ride without the training wheels. For me, this was definitely a highlight of their younger years. Seeing how much they loved it is difficult to describe. Seeing the look on their faces as they realized that they were pedaling all on their own is such a memorable moment. Riding a bike is something I hope will stick with them for the rest of their lives. Every time I see my kids on their bikes, it makes me smile.

To this day, I still ride my bike and I still love it. Though I've moved on to more serious mountain biking and road cycling, part of what I love about it is being outside, seeing the beautiful scenery and getting away from the computer. After all, living in Colorado, I no longer need to look in books and magazines to see such amazing sights. I just go outside and ride my bike to see them.

ActiveMQ In Action Going to Print!

This week we received the final cover art for ActiveMQ In Action and the book went to the printer! Yay!!! It's been a long road, but Dejan, Rob and I can finally celebrate all our hard work on this project.

You can check out the cover art here and be sure to mouse over the image and click the link to view the full size.

There are certainly many people to whom we owe a great deal of gratitude including:

• Jeff Bleiel, our development editor - Without Jeff's input, the book would not be what it is today
• Gary Tully For his tireless technical reviews of the entire book
• The Manning staff - For their arduous work on the book
• Filip Hanik - For his assistance with Tomcat
• Jan Bartel and Greg Wilkins - For their assistance with Jetty
• David Jencks and Kevan Miller - For their assistance with Geronimo
• Jaikiran Pai - For his assistance with JBoss

We would also like to thank the following list of reviewers who read the manuscript at different stages during its development and provided valuable feedback:

• Jeff Davis
• Deepak Vohra
• Robert Hanson
• Davide Piazza
• David Strong
• Tijs Rademakers
• Prasad A. Chodavarapu
• John Merryman
• Jeroen Benckhuijsen
• Pratik Patel
• Scott Dawson
• Jason Kolter
• Rod Biresch
• Roberto Rojas

Finally, a big thanks to the readers of Manning’s Early Access Program (MEAP) releases of the book for their comments and input via the Author Online forum.

We authored the book using DocBook XML and it was processed using the Docbkx Tools Maven plug-in on Mac OS X (To those folks who keep asking about my experience with DocBook, yes, it is worth it! Shoot me an email if you have any questions). Other items that went into the book include MacBook Pros, Google Docs, GMail, Foonz (until it shut down), FreeConferenceCall.com, barking dogs during conference calls, company acquisitions, lots and lots of music, loud construction next door, sleepless nights, too much work on airplanes, and plain old exhaustion.

ActiveMQ and Message Redelivery

Today I helped a customer with a problem in their use of ActiveMQ where messages were getting backed up in a particular queue. It seems that messages in the queue would back up when a downstream process was not available. But this downstream process was only unavailable for a short period of time in the middle of the night for routine maintenance. However, this seems to have caused an ongoing problem for this customer.

We looked at the JVM memory, the broker memory and the queue memory configs each of which was kinda low so I recommended that it be increased (though this was not the problem). I also recommended setting org.apache.activemq.UseDedicatedTaskRunner=false to reduce the number of dispatcher threads being used internally by the broker and to consider the use of the NIO transport instead of the TCP transport to reduce the number of threads used for connections coming into the broker.

As we dug deeper, I looked at the number of consumers on the queue and found that there were 256 Apache Camel consumers in total, none of which was marked as being slow. The consumers were using the ActiveMQ PooledConnectionFactory with the default settings (1 connection and 500 sessions). The consumer prefetch limit was already set to 1 so I knew that this was not a problem. However, while looking at the settings for the connection, I noticed that there was a Camel RedeliveryPolicy set as well. Setting a redelivery policy is not a problem, but the values in it can be a problem if they are not verified.

The settings being used for the Camel RedeliveryPolicy included the following:

• maximumRedeliveries=-1: This means that redelivery attempts will continue forever (i.e., infinite)
• useExponentialBackOff=true: This means that the upon each successive delivery attempt, the amount of delay used will double (because the backoffMultiplier property is set to 2 by default)
• maximumRedeliveryDelay=36000000: This means that the maximum amount of delay that can occur for a given message is 36000000 milliseconds -- yikes that's high! It wasn't until I actually did the math that I saw that 36000000 milliseconds = 10 hours!!!

When you stop and think about the combination of these settings, the problem becomes apparent. Once the exponential backoff causes the maximum redelivery delay for a given message to be reached, that message will be delayed for 10 hours. And if that message experiences another redelivery it will endure the maximum amount of redelivery time again, 10 hours. Geez! (I hazard a guess that the value for the maximumRedeliveryDelay was a simple typo of adding an extra zero. Without the extra zero, it would have been only one hour, not 10 hours.)

I then offered the following advice on dealing with the problem.

1. Using JMX, move the messages from the current queue to a different queue temporarily (let's refer to this different queue as the sandbox1 queue). Make sure that there are no consumers on the sandbox1 queue so you can work on them without a worry that they will be consumed right out from under you.
2. Using Camel, consume the message
3. Using Camel, copy the body and any headers to a new message, but skip the JMSRedelivered header. You may also need to avoid the JMSmessageID header as well since I think it is used to track a message for redelivery. By not copying the JMSRedelivered header on each message it will be stripped.
4. Using Camel, put the message into sandbox2 queue.
5. Manually check each one to make sure that the JMSRedelivered header is actually empty/false.
6. Using JMX, move the messages back to the original queue for the correct consumption and processing.

This is the only way that I can think to escape the 10 hour delay imposed on each message.

How to Access the DocBook DTDs When oasis-open.org Blocks You

Have you ever used the DocBook DTDs (or XSDs) and been blocked by the oasis-open.org website after a handful of accesses to them? If so, here's how to work around this problem. Granted, the solution I offer is specifically for Mac OS X, as long as you can run a webserver on your operating system, the same solution will still work (after all, Mac OS X is really just BSD Unix).

While working on the ActiveMQ In Action book, we were originally using the DocBook 4.5 DTD directly from the oasis-open.org website. Every time I built the book to transform the DocBook XML into PDFs, the Maven build would access the DTDs directly. Pretty standard stuff when the DTDs are not already packaged in a JAR so that they can be accessed locally. I wasn't happy with build ing needing to grab these DTDs for every iteration, but since I have a fast connection it wasn't a big deal for me. However, after a very short period of time, the oasis-open.org website blocked me from accessing the DTDs. This was a pain because it caused the build to fail. To work around this problem, here's what I did.

I simply downloaded the DTDs and related files that I needed, put the DTD and friends in /Library/WebServer/Documents/docbook/xml/4.5/ directory on my local hard drive and added the following entry to the /etc/hosts file so that any requests to oasis-open.org to point to my local machine:

127.0.0.1 www.oasis-open.org

Then I just started up web sharing on the MacBook Pro I used to work on the book and voila! I no longer had to access the DTDs online anymore and the fact that oasis-open.org blocked my IP address didn't matter anymore.

This really wasn't a big deal for me, but I question the motive for making the DTDs publicly available and HTTP accessible if there are rules for accessing them. Why not post work arounds such as this to the oasis-open.org site? Why not just simply post a page of rules for accessing them? Of course, these items may be posted somewhere and I just wasn't able to find them. If so, that's a problem as well. Hint! Hint!

ActiveMQ and Tomcat Articles + 40% Discount on ActiveMQ In Action

As I mentioned here in late October, ActiveMQ In Action entered Manning's pre-production process, the final process of review and editing before actually printing the book. While working through this process, I was asked to write an article for TomcatExpert.com website so I decided to pull a bit of content from one of the chapters that discusses integrating ActiveMQ and Tomcat.

The first article in the three-part series was published titled, ActiveMQ and Tomcat: Perfect Partners. The first article briefly introduces ActiveMQ and the second and third articles provides the details on integrating ActiveMQ with Tomcat using both local JNDI and global JNDI. These two approaches are unique enough and the integration process tricky enough that this info was determined to be good content for an article.

The second article in the series is named Integrating ActiveMQ With Tomcat Using Local JNDI . It covers the integration of ActiveMQ using a local JNDI context. I've had many people come to me who have encountered issues with this style of integration, so I know it can be confusing.

The third article in this series is Integrating ActiveMQ With Tomcat Using Global JNDI and it focuses on configuring JMS resources using global a JNDI context. The global JNDI configuration can be rather difficult because it is a container-wide deployment instead of contained inside of a single web application.

The example application used in the articles comes straight from chapter 8 of ActiveMQ In Action and uses a simple message flow. An HTML form is used to post a message to a queue in ActiveMQ that is consumed by the Spring message listener container. Whereas this example model is simple to demonstrate the power lies in using this model in a more distributed manner and/or with a stand alone ActiveMQ instance. This topic is popular enough that I've been asked numerous times about so I guess it's a good thing it was included in the book.

Just in time for the holidays, you can get 40% off your purchase of ActiveMQ In Action! Just use the coupon code *activemq40* at the time of checkout. Enjoy!

New Features in ActiveMQ 5.4: Automatic Cluster Update and Rebalance

Apache ActiveMQ 5.4.0 was released in August, followed quickly in September by a 5.4.1 release. Not only were there tons of fixes in these releases, but there were also some really great new features including message scheduling, support for web sockets, new Unix control scripts, full message priority, producer message caching and cluster client updates and cluster client rebalancing just to name a few. In this blog post, I'm going to discuss the new cluster client updates and cluster client rebalancing features so that you get a taste of how they are used.

Problem Introduction

When using a network of brokers with ActiveMQ, the configuration of the brokers that form the network has always been rather static. The first step toward a more dynamic network of brokers was a feature I presented in a previous blog post titled How to Use Automatic Failover In an ActiveMQ Network of Brokers. In the event of a broker connection failure, the use of the failover transport for the network connections between brokers allows those connections to be automatically reestablished. This is a wonderful feature for sure, but it only got us part of the way toward a truly dynamic cluster of brokers.

Two new features in ActiveMQ 5.4 introduce the concept of making the cluster of brokers even more dynamic. These two items are the ability to:

• Update the clients in the cluster
• Rebalance the brokers in the cluster

Both of these features are quite interesting so I will explain how each one works.

Update Cluster Clients

In the past, when clients connected to brokers in the cluster, it was recommended to keep a comma-separated list of broker URIs in the failover transport configuration. Below is an example of this style of configuration:

failover:(tcp://machineA:61616,tcp://machineB:61616,tcp://machineC:61616)?randomize=false

The failover configuration example above lives on the client side and contains a static list of the URIs for each broker in the cluster. In the event that a broker in the cluster fails, the failover transport is what allows a client to automatically reconnect to another broker in that list of broker URIs. Unfortunately this style of configuration can be difficult to maintain because it is static. If you want to add another broker to the cluster, every client's failover transport configuration must be updated manually. Depending on the number of clients in your cluster, this could really be a maintenance headache. This is where the first new features comes to the rescue.

ActiveMQ 5.4 provides the ability to automatically update the clients in the cluster. That is, if a new broker joins or leaves the existing network of brokers, the clients' failover transport configurations no longer need to be manipulated manually. Using configuration options on the broker, you can tell the broker to update each client's failover transport configuration automatically. Below is an example of this new feature:


<broker brokerName="brokerA" ...>
...
 <transportConnectors>
   <transportConnector name="tcp-connector" uri="tcp://192.168.0.23:61616" updateClusterClients="true" />
 </<transportConnectors>
...
</broker>


The configuration above is on the broker-side. Notice the new attribute in the <transportConnector> element named updateClusterClients=true. This attribute is used in conjunction with the failover transport on the client-side and it tells the broker to automatically update the client's failover transport configuration when the network topology changes. In addition to the updateClusterClients=true property, there are also a few others including:

• updateClusterClientsOnRemove - Updates a client when brokers are removed from the cluster.
• updateClusterFilter - A comma-separated list of regexes to match broker names that are part of the cluster. This allows flexibility for the inclusion/exclusion of brokers.
• updateURIsURL - Used to provide the path to a file containing a comma-separted list of broker URIs.

These new features are extremely powerful because they allow for a much more dynamic network of brokers configuration. Anyone who has had to deal with the static nature of the failover transport configuration should understand the power in these new features and do some experimentation to see how they operate.

Rebalance Cluster Clients

The second new feature also builds upon the failover transport configuration, but for a slightly different purpose. Consider the fact that when a new broker is added to/removed from the cluster that clients cannot automatically take advantage of it. Even with the new ability to update the clients so that they have knowledge of the broker being added/removed, there was no way previously for them to actually use that broker unless a failure occurred. Well that's what this feature does.

ActiveMQ 5.4 allows clients to be automatically disconnected from their current broker and reconnect to a different broker. Here's an example to illustrate this feature. Let's say you have a cluster of three brokers: brokerA, brokerB and brokerC, each of which has some clients connected. When a new broker is added to the cluster, if the updateClusterClients property is set to true, then the clients will be notified about the new broker, but no action will be taken unless the rebalanceClusterClients property is set to true. When the rebalanceClusterClients property is set to true, the clients will be automatically be disconnected from their current broker in order to reconnect to another broker in the cluster. Below is an example configuration for the new rebalance property:


<broker brokerName="brokerA" ...>
...
 <transportConnectors>
   <transportConnector name="tcp-connector" uri="tcp://192.168.0.23:61616" updateClusterClients="true" rebalanceClusterClients="true" />
 </<transportConnectors>
...
</broker>


Notice the new rebalanceClusterClients attribute in the <transportConnector> element. This property enables the clients to immediately take advantage of the new broker in the cluster. Instead of waiting for the next connection failure and a reconnect from the failover transport, the clients are told to reconnect immediately to another broker in their list.

Testing The New Features

Testing these two new features is pretty easy actually. Below are the steps I have used on a few occasions:

1. Make sure that your clients are logging the broker URI to which they are connected for sending or receiving messages
2. Configure each client to only have one broker URI in its failover transport configuration
3. Configure the transport connector on the broker-side to set the updateClusterClients property to true and the rebalanceClusterClients property to true
4. Start up the brokers in your cluster
5. Start up the clients that connect to a broker in the cluster
6. Add a new broker to the cluster and observe the following behavior:

Due to the two new properties that have been set on the broker-side, each client will be notified of the new broker that was added to the cluster AND each client will automatically reconnect. That is, the functionality of the failover transport will be engaged so that each client is disconnected from the current broker and reconnected to another broker in the list (i.e., the list of broker URIs in the failover transport configuration).

The fact that each client reconnects to a new broker tells you that:

1. The updateClusterClients property is working correctly because you should see the logging change from one broker URI to another. Remember that each client was started with only one broker URI in their failover transport config. The fact that they are reconnecting tells you that they are receiving notifications of changes to the cluster.
2. The rebalanceClusterClients property is working properly because the clients reconnected.

Verify this using the logging from each client. You will see that each client was sending or receiving messages to/from one broker URI and suddenly the logging changes to another broker URI. This tells you that the clients are being updated and rebalanced.

Conclusion

These new features are quite powerful additions to the ActiveMQ network of brokers. They really advance ActiveMQ beyond the static configurations upon which we have all relied for many years now. Most likely the sys admins and dev ops folks will enjoy these features the most because they will no longer need to manually manage a static list of broker URIs for clients.

As I said earlier, many other great features were also introduced in ActiveMQ 5.4 and 5.4.1. So try them out yourself to see if they help to improve your application development.

Update: If you only set the updateClusterClients="true" and the rebalanceClusterClients="true" options, you will notice that when a broker in the network fails and is brought back up, the client connections to other brokers in the network are not automatically rebalanced. This is due to the lack of the updateClusterClientsOnRemove="true" option. After adding this option to the config, network broker clients are notified of broker failures which basically completes the circle and allows the automatic rebalancing to work as it should.

40% Off ActiveMQ In Action To Celebrate Going Into Production!

Good news -- ActiveMQ In Action has now entered the Manning production process! This means that we are done with the writing of content for the book and it is currently being copy edited and undergoing technical review. In fact, we are currently seeking reviewers for the book. If you are interested to review the book, please send an email to mkt@manning.com.

Also, we are currently offering 40% off your purchase of ActiveMQ In Action! Just use the coupon code *activemq40* at the time of checkout. Hurry and make your purchase today as this discount won't last long!