Category: technology
-
How to use librdkafka to build resilient systems
Every good story has a beginning, and beginnings are arbitrary since one thing leads to another, so this story may as well start with a Kafka production incident. The root cause was innocuous, something that was easily fixed (a failed network switch), but since one thing leads to another, one by one, the dominoes fell…
-
New Book – Understanding Message Brokers
If this blog has been unusually quiet over the last few months, it is with good reason – I wrote another book! Understanding Message Brokers has just been released for FREE download by O’Reilly. It is a 70 page concentrated brain dump of how to reason about messaging, based on a presentation I gave at…
-
A Milestone
Last week I completed a consulting engagement with my 100th client since starting work in this field 5.5 years ago. Looking at these last few years in hindsight, it feels like an unfathomable point to be at. I am struck by how much I have learned and how my perspective has developed. It has been…
-
Beyond the Hype Cycle; Co-operative Open Source
At some point in the lifetime of an open-source project, having survived rewrites, deployments in hundreds of organisations, battle-testing… it becomes boring. The sheen goes off it, and people move on to the newer and shinier. When this happens, the number of maintainers drops off, releases slow down to a trickle, and issues or improvements…
-
Performance Test Tool for Apache Kafka
Towards the end of last year, I developed a performance test tool which is available at GitHub for testing the behaviour of Apache Kafka (0.9 so far). The tool was inspired by, and informed by some of the limitations of, the performance tooling in ActiveMQ, which I’m currently retrofitting with the same mechanisms. The kafka-perf-test…
-
Message Distribution and Topic Partitioning in Kafka
When coming over to Apache Kafka from other messaging systems, there’s a conceptual hump that needs to first be crossed, and that is – what is a this topic thing that messages get sent to, and how does message distribution inside it work? Unlike regular brokers, Kafka only has one destination type – a topic…
-
How to reason about queue depth graphs
Occasionally I come across something that seems really obvious to me, but actually isn’t obvious from an outsider’s perspective. A few weeks ago I spent some time with a client (and their client) in trying to understand the behaviour of a message-driven platform that made heavy use of ActiveMQ. Without going into too many details,…
-
Java DSLs for ActiveMQ config and testing
As part of a larger project that I am working on, I needed to define numerous different ActiveMQ configurations inside non-trivial integration tests. Rather than instantiating brokers by hand using the internal ActiveMQ classes, or writing lots of similar-but-different XML configurations, I wrote a pair of Java DSLs to allow me to quickly spin up…
-
Digesting Microservices at muCon
On Friday, I had the privilege of presenting at the very first Microservices conference – muCon. In my talk, Engineering Sanity into Microservices, I spoke about the technical issues surrounding state in distributed systems as a whole, how these become a bigger problem as the number of deployed services goes up, and a few suggested…
-
Running ActiveMQ Replicated LevelDB on VirtualBox
I have wanted to spend some more time recently playing with Replicated LevelDB in ActiveMQ. Not wanting to hit a cloud environment such as EC2, I set up 3 VirtualBox nodes on my workstation running Fedora. The host system needs to be fairly beefy – you need at least a dedicated CPU core per image…