Akka-Streams in practice
Akka-Streams in practice
1/15/18 4:28 PM
|

I've been working on a project with the following back-end technology stack: Scala, Akka-Streams, Akka-Http, RabbitMQ. One of our biggest challenge is to read multiple message queues to consume notification data and send them through the proper push gateway. 

streams

The whole process chain is implemented as a Graph in Akka-Streams like this configuration:

Part1: rabbitSource.acked ~> reportIncomingFlow ~> idempotencyPartitioner.in
Part2: idempotencyPartitioner.out0 ~> dropByIdempotencySink
Part3: idempotencyPartitioner.out1 ~> tokenServiceFlow ~> pushServiceFlow ~> setIdempotencySink

IdempotencyPartitioner forks the chain into two different paths based on the existence(Redis datasource) of the current id of the notification. In case we've got a new id (out1) a tokenService is called, then the pushService is called to deliver the push message to the proper address.

We chose Akka-Streams to get the advantage of built-in buffer management and back-pressure from SInk to the Source. Back-pressure is a technique to tell to the previous Flow element in the Stream if the speed is too fast or too slow. To put it simply this is an architecture where the slowest element of the row tells to the previous ones how fast they should work.

In my Graph, if we read the Stream from its end(setIdempotencySink) to the head(rabbitSource) we understand that the throughput of the whole structure depends on each Flow elements in there. If we could reach the Redis store slow at the end (setIdempontencySink) the previous steps will be slow down in the inner Graph as well like the pushService and tokenService calls. These two services are implemented using Akka-Http to reach out the TokenService API and the Push Gateways. As Akka-Http is developed by Akka as well the Streams API and Http API are working well together.

Developing this infrastructure went well during the first part of this project. Then the first real Push Gateway calls could reach the browsers with the customized messages. Later we had many browser registrations to test how quick the notifications are sent out to different adresses. At one point we realized some of the messages remained in the message queue but the others were delivered. Checked the logs of everything but nothing found at all. No errors logged in the app, nothing found in the queue logs. Debugging the whole Graph from the Source to the Sink didn't come up with any result. It was just broken after a few messages processed successfully.

Playing with the buffer sizes at the Flow elements changed the result. If I raised them high enough, all the messages sent out which got previously stuck in the queue. So I turned my attention to this direction but then I stopped again without any result.

Finally I started to read again the corresponding documentations of the libraries we use from the beginning and I found this in the Akka-Http docs:

"Warning! Consuming (or discarding) the Entity of a request is mandatory! If accidentally left neither consumed or discarded Akka HTTP will assume the incoming data should remain back-pressured, and will stall the incoming data via TCP back-pressure mechanisms. A client should consume the Entity regardless of the status of the HttpResponse."

https://doc.akka.io/docs/akka-http/current/implications-of-streaming-http-entity.html

This became interesting because in some cases - especially when the http request ended with status 200 - we don't care about the http response body itself (reading every time "Push request accepted!") the http response status code is everything we need. Of course if we got any kind of error we would take care of the body for further details. Anyway, reading the meaningless body of each http request solved the issue finally. The queues became empty after fixing it even if I tried with buffer size 1.

I don't know if I learned anything from this troubleshooting session, but what is really annoying about Akka technologies is that it is not easy to debug.