[tech] Building A Notification Service Using Rabbitmq Spring Stomp And Sockjs

This is a technical article on how Acousterr has implemented its notification mechanism.

Background

A notification mechanism is core to any real time web application. Consider Facebook, where we get a notification for likes/comments almost instantly. Acousterr has also built a notification mechanism as there was a need for real time notifications for upvotes, downvotes and comments. The tabs created by any user can be upvoted or downvoted based on which user's ranking and trust among musician community changes. The traditional approach to building such a service would be to poll the server after every few seconds. There is a downside to this approach - there is a significant delay in getting notifications, it is not realtime , and it also puts load on the main API server with constant requests for new notifications, even when the end user is idle. With every request for fetching notifications, a DB call must be made too, which again puts load on the main datastore.

The Solution - Web Sockets

According to wikipedia - "WebSocket is a computer communications protocol, providing full-duplex communication channels over a single TCP connection". Duplex means a two way communication link between client and server, which enables server to send messages to clients as and when a new notification is there. Developers have been using XMLHttpRequest ("XHR") for such purposes, but XHR makes developing web applications that communicate back and forth to the server unnecessarily complex. XHR is basically asynchronous HTTP, and because you need to use a tricky technique like long-hanging GET for sending data from the server to the browser, simple tasks rapidly become complex. As opposed to XMLHttpRequest, WebSockets provide a real bidirectional communication channel in your browser. Once you get a WebSocket connection, you can send data from browser to server by calling a send() method, and receive data from server to browser by an onmessage event handler.

What is STOMP?

STOMP is a simple text-orientated messaging protocol. It defines an interoperable wire format so that any of the available STOMP clients can communicate with any STOMP message broker to provide easy and widespread messaging interoperability among languages and platforms. STOMP is derived on top of WebSockets. STOMP just mentions a few specific ways on how the message frames are exchanged between the client and the server using WebSockets.

RabbitMQ - A Message Broker

RabbitMQ is the most widely deployed open source message broker. Messaging enables software applications to connect and scale. RabbitMQ is a messaging broker - an intermediary for messaging. It gives your applications a common platform to send and receive messages, and your messages a safe place to live until received.


Figure 1 - Simplified overall RabbitMQ architecture. Source: http://kth.diva-portal.org/smash/get/diva2:813137/FULLTEXT01.pdf 

In-memory Simple Message Broker Vs RabbitMQ vs Kafka

Spring does provide an in-memory simple message broker which creates an in-memory message queue and can serve the purpose pretty well for low loads. For Acousterr, initially we had been using this solution as we had a single instance server managing the entire load. But with a multiple instance scenario behind a load balancer, every instance would be having a different in-memory broker,  and notifications could be generating in some other instance, so this solution cannot work in a horizontally scalable application. 

As far as Kafka is concerned, this link can give a comparison of the two

Also from stack overflow - "RabbitMQ is a solid, general purpose message broker that supports several protocols such as AMQP, MQTT, STOMP etc. It can handle high-throughput and common use cases for it is to handle background jobs or as message broker between microservices. Kafka is a message bus optimized for high-ingress data streams and replay. Kafka can be seen as a durable message broker where applications can process and re-process streamed data on disk. Kafka has a very simple routing approach. RabbitMQ has better option if you need to route your messages in complex ways to your consumers. Use Kafka if you need to supporting batch consumers that could be offline, or consumers that want messages at low latency. RabbitMQ will keep all states about consumed/acknowledged/unacknowledged messages while Kafka doesn't, it assumes the consumer keep tracks of what's been consumed and not. RabbitMQ's queues are fastest when they're empty, while Kafka retain large amounts of data with very little overhead - Kafka is designed for holding and distributing large volumes of messages. (If you plan to have very long queues in RabbitMQ you could have a look at lazy queues.) Kafka is built from the ground up with horizontal scaling in mind, while RabbitMQ is mostly designed for vertical scaling. RabbitMQ has a user-friendly interface that let you monitor and handle your RabbitMQ server from a web browser. Among other things queues, connections, channels, exchanges, users and user permissions can be handled - created, deleted and listed in the browser and you can monitor message rates and send/receive messages manually. Kafka manager is yet not as developed as RabbitMQ Management interface. I would say that it's easier/gets faster to get a good understanding about RabbitMQ."

So to summarize - RabbitMQ supports STOMP out of the box, and we had already been using STOMP with in-memory simple message broker. It has a better management interface, simpler to setup and serves our message volumes just fine.

RabbitMQ Setup And Production Tuning

We installed RabbitMQ on an Amazon EC2 instance. Providing relevant commands to install it :

sudo yum -y update
wget http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm
wget http://rpms.famillecollet.com/enterprise/remi-release-6.rpm
sudo rpm -Uvh remi-release-6*.rpm epel-release-6*.rpm

sudo yum install -y erlang

wget http://www.rabbitmq.com/releases/rabbitmq-server/v3.2.2/rabbitmq-server-3.2.2-1.noarch.rpm
sudo rpm --import http://www.rabbitmq.com/rabbitmq-signing-key-public.asc

sudo yum install rabbitmq-server-3.2.2-1.noarch.rpm

sudo rabbitmq-plugins enable rabbitmq_management
sudo rabbitmq-plugins enable rabbitmq_stomp
sudo rabbitmq-plugins enable rabbitmq_web_stomp

sudo rabbitmq-server
#add all ports in security group

sudo rabbitmqctl add_user myadminuser myadminpassword
sudo rabbitmqctl set_user_tags myadminuser administrator
sudo rabbitmqctl set_permissions -p / myadminuser ".*" ".*" ".*"

#start automatically on boot
sudo chkconfig rabbitmq-server on

#start and stop using
/sbin/service rabbitmq-server start

/sbin/service rabbitmq-server stop

or

sudo service rabbitmq-server stop

As seen in above commands, we need to install erlang and rabbitmq-server and add a new admin user, as the default user guest/guest cannot connect from remote server. We we need to enable stomp plugins and optionally the management plugin. For production tuning you can follow this link.

Virtual Hosts

For our case, we didnt need separate virtual hosts for production, development etc as we had a dedicated server for production.

Users

We added a new admin user

Resource Limits and Monitoring

We left all the settings to default values except "open file handles limit", which we increased significantly to allow more concurrent connections. Also in our case the message payload size is pretty small (generally a string enum asking the client to fetch new notifications), so concurrent connections can be increased safely. 

Automatic Connection Recovery

In case the connection dies out, due to either network connectivity or server stopped fr deployment, we need to reconnect. This was handled at client side, where as polling mechanism of reconnection was implemented in case the socket connection is lost.

Cluster size

We have a single instance for RabbitMQ for now

Server Side - Spring Configuration

We need to add a configuration class with @EnableWebSocketMessageBroker annotation, which extends AbstractWebSocketMessageBrokerConfigurer. We need to enable stomp broker relay with 

@Configuration
@EnableWebSocketMessageBroker
public class WebSocketConfig extends AbstractWebSocketMessageBrokerConfigurer {

@Value("${rabbit.hostname}")
String hostname;

@Value("${rabbit.port}")
int port;

@Value("${rabbit.username}")
String username;

@Value("${rabbit.password}")
String password;

@Override
public void configureMessageBroker(MessageBrokerRegistry config) {
config.enableStompBrokerRelay("/queue", "/topic", "/exchange", "/amq/queue")
.setRelayHost(hostname).setRelayPort(port).setSystemLogin(username).setSystemPasscode(password).setClientLogin(username).setClientPasscode(password);
}

@Override
public void registerStompEndpoints(StompEndpointRegistry registry) {
registry.addEndpoint("/notifications-real-time").setAllowedOrigins("*").withSockJS();
}

}

Second we need to add reactor dependencies in pom.xml (or gradle file)
<dependency>
<groupId>io.projectreactor</groupId>
<artifactId>reactor-core</artifactId>
<version>2.0.5.RELEASE</version>
</dependency>
<dependency>
<groupId>io.projectreactor</groupId>
<artifactId>reactor-net</artifactId>
<version>2.0.5.RELEASE</version>
</dependency>
<dependency>
<groupId>io.projectreactor.spring</groupId>
<artifactId>reactor-spring-context</artifactId>
<version>2.0.5.RELEASE</version>
</dependency>
And also spring websocket dependency
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-websocket</artifactId>
<version>1.5.6.RELEASE</version>
</dependency>
And then use SimpTemplate to send messages to a specific topic for the current username. "REFRESH_NOTIFICATIONS" is the payload based on which client can identify the actions to take.

@Autowired SimpMessagingTemplate simpMessagingTemplate;
public void foo() {    
simpMessagingTemplate.convertAndSend("/topic/"+username+ ".notifications", PushMessages.REFRESH_NOTIFICATIONS.name());
}

Client Side Configuration - SockJS

Add the following two scripts in your html file
<script src="https://cdnjs.cloudflare.com/ajax/libs/sockjs-client/1.1.4/sockjs.min.js"></script>
<script src="https://cdnjs.cloudflare.com/ajax/libs/stomp.js/2.3.3/stomp.min.js"></script>
And then handle messages on this usernames topic
window.mysocket = new SockJS('https://www.acousterr.com/notifications-real-time');
window.stompClient = Stomp.over(window.mysocket);
window.stompClient.connect({}, function (frame) {
stompClient.subscribe('/topic/'+username+'.notifications', function (msg) {
if(msg.body == 'REFRESH_NOTIFICATIONS') {
//refetch all notifications
}
});
});

Automatic reconnection :
window.setInterval(function () {
if(window.mysocket.readyState != WebSocket.OPEN)
//reconnect again, using above code
},
10000);

Hope this article was helpful. Do mention in comments if you are stuck somewhere or have any suggestion.
tabs and chords, guitar tabs, music transcription, music technology, deep learning, music information retrieval, transcribe into tabs & chords, chordify, music renditions, songsterr, reverbnation, tabulature, bollywood chords, ultimate guitar, yousician, guitar lessons, acoustic guitar, electric guitar, guitar tuner, 911 tabs