My name is Rex Morgan and I'm a Staff Software Engineer in Austin, TX working at Indeed.

I love learning new languages and tinkering with new technologies. I'm currently having fun with Elixir, Java, C#, and Kubernetes.

Scratching the Itch

A couple of weeks ago I found an article by Ryan Day about implementing the Nasdaq ITCH protocol in python. I was really excited about this because I had always thought the barriers to entry for writing a program relating to the stock market would be very high. Turns out writing an application for stock markets is not the barrier, it's getting the live data needed to be able to make money with it.

I wanted to use RabbitMQ because I've been hearing a lot of great things about it. It has a really nice web interface you can use to check on the status of your queues and see how many messages are passing through. Also, since I had used MassTransit in the past with MSMQ, and MassTransit now supports RabbitMQ, I figured it'd be a breeze to setup.

RabbitMQ was very easy to get up and running, along with the plugin for the web interface. It turns out I don't remember as much as I thought I did about MassTransit, but Joshua Arnold was able to help me get setup pretty quick and I had messages flowing through RabbitMQ in what seemed like no time at all.

The message rate is very slow... I realized that I'm only able to publish about 3,000 messages per second to the message queue. After taking MassTransit out of the mix and publishing directly to RabbitMQ, that number raises to about 10,000 messages per second. If I don't publish any messages at all and only parse the data from the file, I can parse about 450,000 messages per second. So, RabbitMQ and MassTransit are causing a bottleneck.

Consuming messages is even slower. I got MassTransit to pull messages off the queue at a top speed of about 1,200 messages per second. Way too slow to keep up with a trading day, where I think I would need to be able to handle at least 20,000 messages per second in order to keep up with bursts of traffic.

After doing a little digging, I found a nice comparison of some different queues. It sounds to me like ZeroMQ would be a perfect fit for this project. If I have some free time I'm going to give it a shot and see if it performs as well as they say.

Take a look at the code and let me know what you think: http://github.com/RexMorgan/Itch

« More Articles