The main idea of this article is to tell about cases when use of shovel plugin is possible, how to activate and configure it, and also explain the difference between the dynamic and static declaration of shovel plugin.

Use of Shovel plugin

Sometimes a situation may arise when it is necessary to reliably and constantly move messages from the source (for example, queue) of one router to another router (for example, exchange point).

Shovel is a mechanism for transmitting messages from one object (queue) to another. Objects can belong either to one server or different ones. At the same time, shovel plugin may be inactive on the target server, and starting from version 3.7 they can even use different versions of the protocols – AMQP 0.9.1 or AMQP 1.0.

Usage scenario: the need to transfer a message from one application to another, while every application interacts with only one virtual host and does not know anything about the other application.

There are 2 ways to declare shovel: static and dynamic. The differences between them are shown in the table below:

Installation and setup

Installation

To use the capabilities of the plugin, you just need to enable it. To do this you should run the commands from sbin directory:

rabbitmq-plugins enable rabbitmq_shovel rabbitmq_shovel_management

Next, RabbitMQ service will be needed to restart.

After that, new sections will appear in Admin menu:

Setup

It is important to note that the following examples were run on RabbitMQ 3.3.5. Therefore, the names of some variables might differ or mandatory variables might be missing.

Before describing shovel types in details, it’s needed to clarify message acknowledgment modes:

  • on-confirm is the slowest, but also is the most reliable processing mode, as a message is almost impossible to be lost in the event of a malfunction. Deletion (ack) occurs after receipt confirmation at the destination was received (published in the queue);
  • on-publish – “golden mean” in speed and reliability. Deletion (ack) occurs after the message was sent to the destination (published in exchange). Messages may be lost in case of malfunction on the receiving RabbitMQ server;
  • no-ack is the fastest, but at the same time unreliable mode, because the message is deleted immediately without waiting for a response. Messages may be lost due to failures in network infrastructure or RabbitMQ server.

Static definition

In order to create a static shovel, you need to describe it in rabbitmq.config configuration file in rabbitmq_shovel section:

Let’s see what was configured:

  • Lines 299-316 describe the source. In this case, we indicate which exchange point it will be necessary to pick up messages from, as well as the binding of the queue and exchange points.
  • Lines 317-328 describe the receiver. As a recipient, we specify exchange. Please note that in this case, a redirection occurs from one exchange point to another exchange point, but on a different server. It is important to note that if you add {routing_key, << “{{some_routing_key}}” >>} to the publish_fields section, the original routing_key is replaced with {{some_routing_key}}.
  • Lines 329-332 describe behavior of shovel: how many messages can be processed at a time (prefetch_count), confirmation mode (ack_mode), etc.

After restart of RabbitMQ server, you can see that a static shovel has been created that will redirect messages from the local server from Virtual host – host2 to the remote server with Virtual host – host_3. In this case, if there are any problems in configuration, for example, authorization data is specified incorrectly, it will be necessary to restart the server.

Dynamic definition

With dynamic detection, you can create shovel in several ways:

  • using rabbitmqctl utility;
  • using Rest API;
  • through Web UI.

Let’s look at the example of Rest API. To do this, send a PUT request to the address/api/parameters/shovel/{{virtual_host}}/{{shovel_name}}, creating a shovel that:

  1. Will take away messages on the local server on host-host1 from the test1_q queue.
  2. Will send a message to the local server on host – host2, in exchange TEST2.
  3. Reconnection is performed within 10 seconds, 2 messages are taken at a time. Acknowledgment occurs after the messages are confirmed at the destination.

The resulting query body in json format:

{
“value”: {
“src-uri”: “amqp://user@/host1”,
“src-queue”: “test1_q”,
“dest-uri”: “amqp://user@/host2”,
“dest-exchange”: “TEST2”,
“ack-mode”: “on-confirm”,
“reconnect-delay”: 10,
“prefetch-count”: 2
}
}

As a result of the query completion, a new shovel will appear:

It is important to note that with the dynamic definition of shovel, you can look “under the hood” and see what parameters were set in a particular case. Also, if shovel is no longer needed, it can be deleted by pressing a few buttons in the UI without stopping the server.

Conclusion

Shovel plugin is a powerful and flexible tool for sending messages between RabbitMQ servers or within one server. At the same time, it is important to understand that there is no basic task in not redirecting a large flow of messages from the “upper-level” RabbitMQ servers to the “lower” level servers. For this purpose, the Federation plugin is better suited.

When to use a static or a dynamic shovel ad, it is necessary to decide in every single case, if more flexibility in customization and creation speed is required, it is better to use a dynamic ad. If configuration changes are not made and more reliability is needed, then it would be better to use a static ad.

Used links

https://www.rabbitmq.com/plugins.html
https://www.cloudamqp.com/docs/shovel.html