Message queues – implementations and more crazy demands

Hurrah for throwing random ideas around – Simon Wistow has implemented my wouldn’t it be nice.. message queue idea. Regrettably, it’s written in Perl, which means it won’t get any ‘cool kid’ traction and be lost to the mists of history. But good effort anyway, eh?

Though Paul’s comment on that post raises a good point. This stuff belongs in the frameworks nowadays. Sure, there are add-ons. But frankly I want to consider it like the ORM – you can swap out another one if you want, but look, here’s a perfectly good message passing abstraction built right in!

Wait! More ideas! This one has been stewing for longer. It’s a pain writing message handlers in Rails. I want to send email. Or IMs. Or do something useful. But these things are almost always implemented as methods on models, and in the model I’m not supposed to know about the outside world. In rails, models don’t know how to generate urls, for instance (whereas in Django, models are responsible for knowing what their authoritative representation URL is. This is alternately great and weird to me. How do I have a different URL for the object on the mobile site?).

The interface I really want is the standard MVC model, where in this case, the View is the message transport layer, and my message handling code is in the Controller. I want access to my url helpers, so if I proxy incoming Jabber messages to queue messages (which is probably a good idea) I can reply to these messages and link to web pages. I want a session abstraction, tied to the transport layer, so I can fake a long-running conversation, and I want it to look like the web session abstraction. Sure, I can do the session stuff myself, but it belongs in the transport layer abstraction – noone would seriously suggest you write your own web session handler nowadays, would they? The ‘web’ interface to my app shouldn’t be ‘special’ like this.

Image by bartmaguire

Faking a message queue

Right. Here’s an idea I want someone else to build, because I’m too lazy/busy. I want a Stomp interface to mysql.

Let me explain. There are lots of reasons you might want a message queue architecture in your new web application. You want long-running background processes, or you want to fan out changes to lots of users without tying up the web server process. Or you want to make calls to an API somewhere, again without tying up an expensive web server process. It’s a lot easier to build this stuff in from the beginning than to try to work out later which bits of your application flow don’t need to happen before the web page is returned to the user.

At the same time, it’s a tiny new web site. You don’t want to have to run a message queue just yet, so you want to fake it – write tasks into a database table, and have a long-running daemon loop forever, pulling out rows in order and doing them. Almost as good as a real message queue for small volumes of messages, and a lot cheaper. So what I want is a way of doing this that has the same API as a real message queue, so that when you cut over, you don’t have to change any code, everything just gets more responsive. Yay!

Right. Now someone do it. Let me know when you’re done.