Home >  Archive >  2012 >  September >  5

Message based web applications

Every time I encounter a hard software problem I usually end up using a message queue to solve it. Workers grab messages from the queue and process them in a controlled manner. Whatever chaos there was before is now under control. permalink

A few months ago I had a simple thought: if queues are so great at solving all of these problems why not use them all the time? permalink

What if the entire system was a series of message queues and workers? permalink

This idea dovetailed with my recent interest in non-blocking servers and adopting a callback mentality when developing the original FANS.FM codebase in node.js. My software mind has been trending to the asynchronous lately so why not take it a step further. permalink

The benefits could be huge. permalink

A message based system could be comprised of worker scripts written in whatever language you'd like. The queue is language neutral and as long as the messages were in a standard format any script could handle the work to be done. This means I could mix node.js, Ruby and Java seamlessly within the same system. permalink

Work can be guaranteed. A true MQ system will offer message acknowledgement features to ensure that a message was processed by a worker. If a worker fails, is restarted or killed the messge will be requeued and not lost. permalink

Scale could be achieved on a fine-grained level. If a particular queue gets a lot of traffic assign more workers to it. permalink

Code could be deployed in realtime. Since messages are always persistent no work will be lost. Worker processes can be quickly restarted and the system can effectively achieve no absolute downtime. permalink

Message based systems have been around forever. The way I'm applying the concept is what's new and interesting (at least to me). permalink

If you read this far and are lost: don't feel too bad. This isn't the greatest explanation. I'll diagram out the queues and message flow in a future post. permalink

I am now testing this theory out in my re-factoring of FANS.FM. I'll let you know how it goes. permalink