Messaging systems - practical guidance

Have you ever faced the problem of attaching a queueing system to an existing project? Are you a programmer or a product owner wondering if it’s worth investing time and resources in the implementation of such a system? If so, I come to your aid. In this article I will be walking you through the specific use case of using a queuing system as a technical solution for a single feature in a medium size app.

Finding basic information about what queuing systems are and what components they consist of is fairly simple - it’s understanding the practical application that takes more time.

For this reason, in this two-part guide, I would like to analyze the pros and cons of messaging systems in the context of your application. I would like to consider what you will have to deal with during implementation and also tell you about the cases in which you definitely should not use messaging systems.

The first part focuses on challenging your brain with the important questions you have to ask yourself when considering the messaging system. In the second part, I present what problems I had to face when implementing such a system in one of the applications I created. In the end, I will also tell you if I made a good decision and why or why not.

After reading this article I want you to be able to evaluate:

• whether a notification system is something you should use in your application,
• whether the costs you have to bear are less than the potential profit from the solution.

But what is a message queue?

In a nutshell, a message queue is a tool for asynchronous processing - it buffers and distributes asynchronous requests.

In a nutshell, a message queue is a tool for asynchronous processing - it buffers and distributes asynchronous requests. Requests and messages - usually a piece of XML or JSON - are created by message producers, sent to the queue, where they are buffered, and afterwards delivered to the consumer. Producers and consumers are usually separate processes which work independently of each other and are coupled only by the message format and infrastructure constraints like message queue location. The heart of the messaging system is the queue, usually referred to as a message broker. In its simplest form, it can be a separate thread running inside the main application process, but usually, a message broker is a complex application responsible for fast and reliable processing of messages with specialized features like routing, permissions control, or failure recovery.

Now, when you more or less know what the messaging system consists of, let’s start discussing our main thread.

Do I really need this queue?

You can definitely repeat some facts about the advantages of queuing systems: they facilitate scaling, promote efficiency, allow decoupling, and are fantastic in terms of system reliability. But do you really need all these features in your system and what are the costs of introducing them?

At any point in the future when you’re designing or exploring possible solutions, you will face the problem of deciding whether to use a message queue or to implement something differently. In this scenario, you should look back on the questions below and evaluate them with your team.

1. What kind of processes can I identify in the feature which I’m designing?

Imagine that you are designing a new feature. There are many different types of actions(handling requests, database operations, parsing and transforming data, publishing events, etc.) which your application performs to handle the business logic needed for meeting the requirements of the feature correctly. Think of:

• when you execute a specific action,
• what is the role of this specific action,
• whether you can postpone the execution of the action or not,
• whether the action is crucial in the flow of the algorithm,
• whether the action necessarily needs to be completed in order to return a response to the user,
• whether it is highly probable that the number of actions to handle can grow unpredictably,
• how much resources a specific action needs.

Thinking of the actions in such a way will allow you to identify the type of actions that are good candidates for processing via a message queue. At least three groups of such tasks exist:

• resource-intensive tasks - such tasks very likely require a bit more time to be finished, which will probably have a significant impact on the overall performance of the application. This group can contain actions like PDF creation, processing images or videos, generating reports, etc.
• remote server calls - such operations often can be deferred. There are many cases when you don’t need to wait for the response from the remote server. You can just schedule the execution of the action and forget about it. The results of the actions performed somewhere can be displayed later. Imagine the tasks like: send a confirmation email after a purchase, add a client to the subscribers’ list in the external marketing system, and so on.
• tasks non-crucial in a business context - very often the requirements that we have to meet within a given functionality can be divided into two groups: critical, i.e. the ones that have to be met at a given moment, and side ones, the ones that you often wonder about: do I really have to do it at this moment? For example, think about the functionality of adding a product to the store. What is crucial here: creating a record in the database and informing the user that the product has been added. All the other activities: saving and processing (e.g.: resizing or saving several sizes) of the product's photos, updating the external shipping system, re-indexing the advanced product search engine, etc. are all side tasks. Of course, all of them are also super important, but technically we can just do them later.

2. How many operations of the specific type I will have to perform?

You already know that you will implement queuing tasks to meet the requirements of the functionality. It is important to note that just because these tasks exist doesn't mean that you have to use the queue to handle them. Other questions that are worth asking are in the context of the number of operations performed:

• how many operations of a given type I will have to handle and what is the predicted amount of time the single operation will consume,
• whether the number of actions to be performed will be more or less constant or whether I will have to handle sudden, unexpected spikes (increases) in the number of operations,
• whether the number of operations will increase in time, and if so, at what rate.

Each of these questions is related to two important features of queued systems: easier system scaling and the ability to even out the level of traffic (and thus the resources used) over the lifetime of the application. Applications with message brokers are usually easier to scale thanks to the publish-and-forget mechanism. Your application can just quickly publish a message, return a response to the client and handle another request.

Applications with message brokers are usually easier to scale thanks to the publish-and-forget mechanism. Your application can just quickly publish a message, return a response to the client and handle another request.

In the meantime, somewhere in the background multiple parallel processes can listen to the queue and handle the operations. If suddenly you will have to handle more requests, then you just add another consumer that will handle these additional messages. Thanks to the message queue, parts of the application are decoupled and on the publisher side, you don’t have to worry about changing the configuration or performing any type of work.

How to interpret the answers to the questions above? Simply: when you have many operations to perform where execution takes significant amounts of time and the number of operations can constantly (or temporarily but suddenly) grow - then you definitely should implement the message queue in your application and you will reap the rewards. In any other case, unfortunately, it depends - on the initial amount of operations, the rate of your application’s growth, the business circumstances, who is responsible for the infrastructure... There is no simple, single answer here.

Let’s move on to the next question which can help us to finally decide. Maybe other attributes will have a significant impact on the final verdict?

3. How reliable does the system have to be?
There are a couple of things to consider in relation to the reliability of the system. Remember what you read above about tasks that are not-crucial in the business context. Think that every step which is not necessarily needed on the critical path is a potential point of failure. Thanks to the message queue we’re able to move the point of failure to external apps which are independent from the main process, making the overall system fault-tolerant. It also works the other way around. Consumers are independent from the producer (which handles the critical path) and in case of failure of the producer, consumers can still normally work as long as there are messages in the queue. Additionally, message queue features such as “guaranteed delivery” allow us to assume that the message will be processed correctly nearly always.

So... Do I ultimately need this message queue or not?

Summarizing this part of the discussion. Unfortunately, as always, there is no single, clear-cut answer. I hope that answering the questions above can at least help you better understand your real needs and prompt you with some conclusion. The final decision it’s up to you.

Obviously the advantages shown above are only the single side of the problem. I would like to invite you to the second part of this debate, where I will tell you more about the message queue related challenges and where we make a broader summary.

Appendix / Bonus

Take a look also at the “matrix” below, these “yes/no” questions can help you even more (you can also add some other questions which will cover your specific case or constraints you have). When you will answer “yes” for all of these questions then you should definitely resolve your problem with the message broker somewhere in the background or at least treat such a solution as one from the most proper. But if for any of the questions you say “no”, then you have to go deeper and understand the cons of the messaging systems and the problems which this solution brings you sooner or later.

Looking for an experienced team to bring your digital product to life?

Get in touch for a free consultation on hello@10Clouds.com. Our friendly team will get back to you within one working day!