Networks of message brokers in ActiveMQ work quite differently to more familiar models such as that of physical networks. They are not any harder to understand or reason about but we need to have an appreciation as to what exactly each of the pieces in the puzzle do by themselves in order to understand them in them large. I will try to explain each component piece progressively moving up in complexity from a single broker through to a full blown network. At the end you should have a feel of how these networks behave and be able to reason about the interactions across different topologies.
One of the key things in understanding broker-based messaging is that the production, or sending of a message, is disconnected from the consumption of that message. The broker acts as an intermediary, serving to make the method by which a method is consumed as well as the route that the message has travelled orthogonal to its production. You shouldn’t need to understand the entire postal system to know that you post a letter in the big red box and eventually it will arrive in the little box at the front of the recipient’s house. Same idea applies here.
Connections are shown in the direction of where they were established from (i.e. Consumer connects to Broker).
Out of the box when a standard message is sent to a queue from a producer, it is sent to the broker, which persists it in its message store. By default this is in KahaDB, but it can configured to be stored in memory, which buys performance at the cost of reliability. Once the broker has confirmation that the message has been persisted in the journal (the terms journal and message store are often used interchangeably), it responds with an acknowledgement back to the producer. The thread sending the message from the producer is blocked at this time.
On the consumption side, when a message listener is registered or a call to receive() is made, the broker creates a subscription to that queue. Messages are fetched from the message store and passed to the consumer; it’s usually done in batches, and the fetching is a lot more complex than simply read from disk, but that’s the general idea. With the default behaviour, assuming the Session.AUTO_ACKNOWLEDGE is being used, the consumer will acknowledge that the message has been received before processing it. On receiving the acknowledgement, the broker updates the message store marking that message as consumed, or just deletes it (this depends on the persistence mechanism).
If you want the consumer to acknowledge the message after it has been sucessfully consumed, you need to set up transaction management, or handle it manually using Session.CLIENT_ACKNOWLEDGE.
So what happens when there are more than one consumer on a queue? All things being equal, and ignoring consumer priorities, the broker will in this case hand out incoming messages in a round-robin manner to each subscriber.
Store-and-forward
Now to scale this up to two brokers, Broker1 and Broker2. In ActiveMQ a network of brokers is set up by connecting a networkConnector to a transportConnector (think of it as a socket listening on a port). A networkConnector is an outbound connection from one broker to another.
When a subscription is made to a queue on Broker2, that broker tells the other brokers that it knows about (in our case, just Broker1) that it is interested in that queue; another subscription is now made on Broker1 with Broker2 as the consumer. As far as an ActiveMQ broker is concerned there is no difference between a standard client consuming messages, or another broker acting on behalf of a client. They are treated in the exact same manner.
So now that Broker1 sees a subscription from Broker2, what happens? The result is a hybrid of the two producer and consumer behaviours. Broker1 is the producer, and Broker2 the consumer. Messages are fetched from Broker1’s message store, passed to Broker2. Broker2 processes the message by store-ing it in its journal, and acknowledges consumption of that message. Broker1 then marks the message as consumed.
The simple consume case then applies as Broker2 forwards the message to its consumers, as tough the message was produced directly into it. Neither the producer nor consumer are aware that any network of brokers exists, it is orthogonal to their functionality – a key driver of this style of messaging.
Local and remote consumers
It has already been noted that as far as a broker is concerned, all subscriptions are equal. To it there is no difference between a local “real” consumer, and another broker that is going to forward those messages on. Hence incoming messages will be handed out round-robin as usual. If we have 2 consumers – Consumer1 on Broker1, and Consumer2 on Broker2 – if messages are produced to Broker1, both consumers will each receive the same number of messages.
A networkConnector is unidirectional by default, which means that the broker initiating the connector acts as a client, forwarding its subscriptions. Broker2 in this case subscribes on behalf of its consumers to Broker1. Broker2 however will not be made aware of subscriptions on Broker1. networkConnectors can however be made duplex, such that subscriptions are passed in both directions.
So let’s take it one step further with a network that demonstrates why it is a bad idea to connect brokers to each other in an ad-hoc manner. Let’s add Broker3 into the mix such that it connects into Broker1, and Broker2 sets up a second networkConnector into Broker3. All networkConnectors are set up as duplex.
This is a common approach people take when they first encounter broker networks and want to connect a number of brokers to each other, as they are naturally used to the internet model of network behaviour where traffic is routed down the shortest path. If we think about it from first principles, it quickly becomes apparent that is not the best approach. Let’s examine what happens when a consumer connects to Broker2.
- Broker2 echoes the subscription to the brokers it knows about – Broker1 and Broker3.
- Broker3 echoes the subscription down all
networkConnectors other than the one from which the request came; it subscribes to Broker1. - A producer sends messages into Broker1.
- Broker1 stores and forwards messages to the active subscriptions on it’s
transportConnector; half to Broker2, and half to Broker3. - Broker2 stores and forwards to it’s consumer.
- Broker3 stores and forwards to Broker2.
Eventually everything ends up at the consumer, but some messages ended up needlessly travelling Broker1->Broker3->Broker2, while the others went by the more direct route Broker1->Broker2. Add more brokers into the mix, and the store-and-forward traffic increases exponentially as messages flow through any number of weird and wonderful routes.
Fortunately, it is possible to avoid this by employing other topologies, such as hub and spoke.
You can also use a more nuanced approach that includes considerations such as unidirectional networkConnectors that pass only a certain subscriptions, or reducing consumer priority such that further consumers have a lower priority than closer ones.
Each network design needs to be considered separately and trades off considerations such message load, amount of hardware at your disposal, latency (number of hops) and reliability. When you understand how all the parts fit and think about the overall topology from first principles, it’s much easier to work through.
Update (July 2014): To get a better understanding of what’s going on under the covers, check out the details of how network connectors work.
Comments
23 responses to “Understanding ActiveMQ Broker Networks”
Your post is really doing a great job explaining activemq network!
[…] Jakub wrote an excellent blog post explaining more on how ActiveMQ networks work. The one things that somehow always got unexplained […]
Thanks for the step-by-step description. Really helps to approach it like that when dealing with networks of brokers that can get complicated.
In your last diagram, what happens if BrokerHub crashes? Doesn’t the whole thing fail? How do you introduce redundancy into this?
That architecture definitely introduces a central point of failure. You can mitigate this by replacing that logical broker, and indeed the other individual brokers, with highly available master-slave pairs to add a level of reduncancy.
When a subscription is made to a queue on Broker2, that broker tells the other brokers that it knows about (in our case, just Broker1) that it is interested in that queue; another subscription is now made on Broker1 with Broker2 as the consumer. As far as an ActiveMQ broker is concerned there is no difference between a standard client consuming messages, or another broker acting on behalf of a client. They are treated in the exact same manner.
So now that Broker1 sees a subscription from Broker2, what happens? The result is a hybrid of the two producer and consumer behaviours. Broker1 is the producer, and Broker2 the consumer. Messages are fetched from Broker1′s message store, passed to Broker2. Broker2 processes the message by store-ing it in its journal, and acknowledges consumption of that message. Broker1 then marks the message as consumed.
Can you explain this part more clearly??
Jake, the HA master-slave really doesn’t do what I want because, as I understand it, the slave sits there doing nothing until the master dies. I’m really having trouble getting a grip on how you have all servers in the cluster doing work while achieving high availability. It seems like I’d need to do exactly what you’re advising against in this post. Not your fault, of course, but if I’m right in what I’m saying the AMQ design for clustering is really not very well-thought through. Thanks for you post, though, and your responses.
Mark, you’re correct in saying the slave does nothing. The idea is that one node from the master-slave pair is the available master, taking inbound connections. The pair is treated as a single logical broker. So if you wanted the hub to be highly available on two hosts, hostA and hostB, all incoming networkConnectors would be configured as “static:(tcp//:hostA:portA,tcp://hostB:portB)”. This way if one of the boxes that serves the logical broker goes down, the other picks up from where the box that went down left off. The same setup applies to most broker network topologies, where each critical logical node is actually two HA brokers on seperate servers. If you want every box to be doing work, there are ways of achieving that using things such as eventual consistency, but those are beyond the scope of my blog post and would need a book to themselves, as their application varies widely depending on the details of the use case. If you are working on a broker network and would like a hand with the details, the company I work for, FuseSource, does ActiveMQ consulting.
[…] Jakub wrote an excellent blog post explaining more on how ActiveMQ networks work. The one thing that somehow always got unexplained […]
You obviously know what you are talking about!! This is a +1 for this page… You made understanding the rather convoluted matter of setting up multiple brokers straightforward.
My scenario : The “plugins” capability is broker wide – and there is no support to configure per transportConnector authentication : http://activemq.2283324.n4.nabble.com/Per-transportconnector-authentication-td3752107.html
I really needed the external broker to consume messages, traffic created local to my ecosystem of producers that are all local to my VM. But then I was unit testing my code, I couldn’t figure out for the love of God – why messages produced on the remote broker were not being propagated to the networked broker, and were just routed back to consumers configured on the remote broker! The term “outbound connection” made it very clear, which made me look at and consider the “duplex” attribute more closely. Since I only have 2 brokers, it might not be terrible to have the networkConnector run in duplex mode.
The only pitfall of my approach to achieve the authentication scheme that I desire was that I had to configure 2 separate data stores for the brokers, as brokers cannot share databases – it is a good workaround, until per-transportconnector picks up some interest. Perhaps a TransportConnectorFilter implementation is in order, but I am not holding my breath for it!
You mentioned why a fully meshed network is not a good idea. But
can’t we use a fully meshed network of brokers with each network
connector with a networkTTL of 1. Wouldn’t it address the issues
you were referring to.
Unfortunately, that wouldn’t work. Setting that would mean that the messages can only travel as far as the next broker before they need to be consumed directly from there (they won’t move down any other networkConnectors beyond that). So unless you have a consumer that’s able to consume messages sitting on that broker, you would most likely see messages being “stuck”.
[…] https://www.jakubkorab.net/2011/11/understanding-activemq-broker-networks.html […]
thanks for explanation, i was so confused with network brokers
concept and was struggling with it for last 3 days. thanks a lot 🙂
Thanks for the post! I was wondering how does the networking goes if you are using a topic subscription (pub-sub)
Nice article.
Thanks for the clear explanation.
Hi Jakub, first of all this should become the official documentation. Great Post, mate!
Now, let me seize the opportunity and ask you a question… I hope you can clarify me something.
We have 3 nodes and on each one we have one MASTER broker and one SLAVE broker (for a master on other machine).
NODE1 MASTER1 SLAVE2
NODE2 MASTER2 SLAVE3
NODE3 MASTER3 SLAVE1
When we start the brokers we see that MASTERS are up and listening their ports and slaves are started, but down/inactive. Then in one moment SLAVE starts listening its port although its master is UP too. When this scenario occurs we start loosing some of the messages. We use Oracle for the persistence of messages where each MASTER has its Oracle user_masterX and each slave has its user_slaveX Oracle user.
Looking more closely we see that the messages end up in Oracle table ACTIVEMQ_MSG, but in the tables slave broker has created that cannot be seen by the MASTER. It seems like the producer sends messages to the broker and they end up in its slave’s tables. This way the consumer doesn’t see any new messages on the MASTER it’s listening…
Hope you will be able to help me even without any config. details 🙂
Thank you for this post anyway!
I found this very resourceful, many thanks for sharing.
Also, this is the networkConnector configuration from activemq.xml
(BrokerA)
(BrokerB)
Again, thank you in advance if you can shed some light into my problem.
(Edit on my previous post)
(Broker A)
networkConnector name=”linkToBrokerB”
uri=”static://(tcp://10.244.7.30:61616)”
networkTTL=”3″
userName=”admin”
password=”password”
conduitSubscriptions=”false”
duplex=”true”
networkConnectors
transportConnectors
transportConnector name=”openwire” uri=”tcp://0.0.0.0:61616?maximumConnections=1000″/
transportConnectors
(Broker B)
networkConnectors
networkConnector name=”linkToBrokerA”
uri=”static://(tcp://10.244.7.29:61616)”
networkTTL=”3″
userName=”admin”
password=”password”
duplex=”true”
networkConnectors
transportConnectors
transportConnector name=”openwire” uri=”tcp://0.0.0.0:61616?maximumConnections=1000″
transportConnectors
[…] you are prepared to use ActivaMQ in Eclipse Virgo. This is a really good article talking about the mechanism of […]
Where can I find a dirt simple manual to configure two machines to
send and receive to each other? I have looked at Apache website
but it gets very technical very quick. I have ActiveMQ on two web
servers. I want to use PHP top send message from machine a to
machine b. How can I set up Active MQ on both sides to do this?
Well explained, thanks.