Dear Lazyweb,

I'm interested getting an idea of what open source message queueing systems exist that are fast, stable, and have some good replication (think multi-colo) and fault tolerance built-in. The idea being, of course, that some processes want to send messages into a queue (of work to be done) and other processes will fetch those and do stuff with them.

Ideally, I'm looking for a system that allows for different message priorities--meaning that I'd like to be able to mark some messages as less important, so it's okay if we lose them in a crash. It'd also be handy to have the ability to set expiation times on messages.

Bonus points for stuff with good Perl libraries.

Put another way, if you wanted to run something like Amazon's SQS on your own infrastructure, what would you use as the building blocks?

Stuff I already know of (some of which doesn't meet my own criteria):

But surely there's more. Feel free to spew others in the comments below...

And even if you don't know of any others, I'd love to hear about your experience with any of the above or already commented systems.

Update: A lot of folks are replying with "what's wrong with XXX in your list?" I haven't tested these yet. I'm looking to see what the landscape looks like before I dive in.

Posted by jzawodn at August 18, 2008 11:49 AM

Reader Comments
# Kurt said:

RabbitMQ is perfect because it's written in Erlang. Right!?

on August 18, 2008 12:24 PM
# Dave said:

I think the hardest requirement is: fault tolerance built in. You need to carefully define what that means. For example, some software can actually duplicate each message so that if a provider fails, the message can still be delivered. Others allow two providers to share a disk.

on August 18, 2008 12:40 PM
# Jason said:

There's also zeromq [http://www.zeromq.org/], which I came upon recently thanks to Delicious.

on August 18, 2008 12:40 PM
# Michael Buckbee said:

For the "some messages of less importance" requirement, I think you could technically do this with any of the systems by declaring a different queue. This would also seem to fit better with your example of losing them in a crash: put the low priority stuff on the less fault tolerant system.

Also, can you share any thoughts on why you wouldn't just go with SQS?

on August 18, 2008 12:57 PM
# said:

http://pi3.informatik.uni-mannheim.de/demaq.html.en is a freely available (though not open sourced) messaging system built on a native XML-Database. Actions on messages are specified in a declarative Xquery like fashion.

on August 18, 2008 01:00 PM
# Guillaume Theoret said:

I also agree that you need to define what you mean by fault tolerance.

Do you mean that if the plug gets physically pulled on a server that the producers automatically send to different servers and, when that server comes back online, none of the jobs it had in memory are lost?

If so, then I recommend RabbitMQ. Most use cases can be handled by basicPublish which supports priority queues (As shown in the signature here: http://www.rabbitmq.com/api-guide.html#publishing )

I'm recommending RabbitMQ over OpenAMQ and Qpid because they seem to have better/more documentation and an active community. Theoretically though you should be able to swap out RabbitMQ for either OpenAMQ or Qpid and your clients would keep working though. (In practice however there are still small incompatibilities in advanced features but it's close to being transparent)

I'm not sure about Perl bindings for AMQP though. I'm using the excellent Java client RabbitMQ provides but I did notice talk about a perl module for AMQP: http://www.nntp.perl.org/group/perl.modules/2008/06/msg60950.html

All this is if you need the server to keep its messages if the plug is pulled. If that's not a requirement (if an in-memory queue is fine), I'd suggest you look into beanstalkd which has a perl client library: http://xph.us/software/beanstalkd/client.html Most libraries implement dynamic failover I believe (but I'm not 100% on so you should check before believing me!).

on August 18, 2008 01:15 PM
# Dirkjan Ochtman said:

Well, I'm partial to XMPP just because it's an IETF RFC and I've played with it and it's readable and extensible and seems generally nice. Since you seem to be looking for Perl, maybe djabberd is a good fit? I seem to remember Brad Fitzpatrick (of LiveJournal fame) also wrote some other Perl queueing software, though I don't remember what it was called. Gearman, or some such? Maybe perlbal was also part of it.

on August 18, 2008 01:27 PM
# Elias Torres said:

I did a little survey on the subject and OpenMQ from Sun it's a little gem.

https://mq.dev.java.net/

on August 18, 2008 01:28 PM
# Raoul Duke said:

Yeah, ZeroMQ seems to be the latest kid on the block, but I have no experience with it. The site's claims definitely make my mouth water.

on August 18, 2008 01:38 PM
# Serg Gulko said:

Dear jzawodn!

Right now we are working under Perl Client library for RabbitMQ. I cant tell you final date of release but for sure we will do it.

From my previous experience... We was using Spread alot with Perl and this messaging system show himself from best side. It very fast, stable and easy to use. But:)! Spread are...how to say more correct...he is low level. You need to care about almost all things(except multicasting of course:)).

RabbitMQ from this point of view providing much more advanced services. Actually, this advantages cause us start working under Perl Client.

Serg Gulko

on August 18, 2008 01:45 PM
# pwb said:

SMTP + POP/IMAP.

on August 18, 2008 02:29 PM
# Andrea Baron said:

I'm not into this, but I heard in this podcast:
http://deepfriedbytes.com/podcast/deep-fried-bytes-episode-4-scaling-large-web-sites-with-joe-stump-lead-architect-at-digg/
at digg they're using Gearman:
http://www.danga.com/gearman/
it's not reliable, but looking a little further LiveJournal uses TheSchwartz
http://search.cpan.org/~bradfitz/TheSchwartz-1.07/lib/TheSchwartz.pm
with it to make a reliable MQ system.

on August 18, 2008 02:45 PM
# Serg Gulko said:

We tried Gearman too... But for pure communication(data transmission) I'd recomend use Spread.

on August 18, 2008 03:09 PM
# Pedro Melo said:

Hi,

two systems: TheSchwarts (Danga.com or CPAN, but I recommend the SVN version), and SAPO Broker (http://softwarelivre.sapo.pt/broker/).

Best regards,

on August 18, 2008 03:55 PM
# John Herren said:

I've used both Spread and SQS, but not with the same requirements. Spread gave me a constant 1% cpu utilization, but it is supposed to have guaranteed message delivery. I did notice that some of the Perl clients I wrote to read the Spread stream would flake out sometimes under very high load. I'm sure that was my fault, though.

Regardless of what you go with, please be sure to write a follow-up and share with us what works best for you.

on August 18, 2008 04:32 PM
# Dave Hodson said:

Take a look at JBoss MQ (or whatever they are now calling it). I used it on a prior project that is now pulling over a billion transactions/month. Overall, I found it to scale quite well.

on August 18, 2008 05:57 PM
# kellan said:

So, having evaluated those toolkits what didn't you like about them? The list of issues would be fascinating.

Personally I've been playing with RabbitMQ and am pretty happy with it. Stayed away from ActiveMQ mostly based on hearsay and horror stories.

on August 18, 2008 06:06 PM
# jarjar said:

check out http://code.google.com/p/sparrow/

you can use any of the perl memcached client libraries to access it.

on August 18, 2008 06:47 PM
# blckholehorizon said:

OpenMQ has quite sparked my interest personally as something for use in OSS, but how does it compare to the commercial {& expensive} package BEA Message Queue? or its commercial package Java System Message Queue? I've used BEA quite a bit in a manufacturing setting, and it has never failed me { /yet/ crossing fingers }. BEA definitely provides a level of service for rock solid messaging in our closed source software systems.

Being a avid supported of OSS, how can we deploy something like Open Message Queue. Being a young software engineer, I'm trying to learn the in's and out's of the plethora of licensing models out there.

We've definitely tried to roll our own "Simple" Messaging Subsystem in our group, and never really ended up with what we needed, or something flexible enough to use again. It always ended up going in the direction of something that resembled a depraved version of BEA's little brother which was much wasted effort.

But not having the correct mindset from management, nor budget allocation puts us in quite the predicament. So I guess I'll just have to pick my poison from what's out there that fits my situation.

http://www.computer.org/portal/site/dsonline/menuitem.9ed3d9924aeb0dcd82ccc6716bbe36ec/index.jsp?&pName=dso_level1&path=dsonline/topics/middleware&file=projects_MOM.xml&xsl=article.xsl&;jsessionid=LqqTjmvSygDp5kspdlv027zSB671S15ZnQdykLl2CkzwmGQ3Q2bQ!-2139236801 is a decent list of some others that are out there...

I may have to play around with RabbitMQ myself here in the next few days to see what it can provide.

on August 18, 2008 07:12 PM
# Justin Mason said:

Hi Jeremy --

Gearman is nice, fast, perlish and reliable -- we are now using it on the SpamAssassin spamtraps with great results. It's by Brad Fitzpatrick, his stuff is always solid and good perl. It also supports "high priority" messages which skip the queue. However, it doesn't persist messages to disk :(

I understand TheSchwartz is what LJ use to add a more-reliable fault-tolerance layer, but it trades off speed for fault-tolerance. I haven't tried it.

Most of the other new queueing systems are fast, but seem to be lightweight and non-persistent. :(

I have plans to use Gearman soon in another project, with persistence -- I'll add a backup layer of SQL-hosted persistence, myself, to allow recovery of the Gearman queues if the gearmand winds up dying/rebooting etc.

(hey btw I notice that my "via:jzawodny" tag on del.icio.us is now the top hit for "jzawodny" ;)

on August 19, 2008 02:24 AM
# Erik Weibust said:

What was the problem with ActiveMQ? Without cross-checking each of your requirements, I think it would work.

on August 19, 2008 09:51 AM
# Josh Wand said:

Twitter wrote:

http://rubyforge.org/projects/starling/

"a light-weight persistent queue server that speaks the MemCache protocol. It was built to drive Twitter's backend, and is in production across Twitter's cluster."

on August 19, 2008 11:04 AM
# Arnaud said:

Hello,

The following may prove interesting

"JLog is short for "journaled log" and this package is really an API and implementation that is libjlog. What is libjlog? libjlog is a pure C, very simple durable message queue with multiple subscribers and publishers (both thread and multi-process safe)" ...

I have never used it but it seems to have good perl bindings

https://labs.omniti.com/trac/jlog

on August 19, 2008 01:49 PM
# Jonathan Disher said:

I was just looking at this a couple of weeks ago - it looks like we have ActiveMQ in dist here at Y!, I'm not aware of who's using it. I personally am looking at using it in my property, it's open source, and appears to have good php and perl bindings, in addition to the usual JMS, etc.

http://activemq.apache.org/

Good luck, if you find anything better I'd love to hear about it. Hope Craigslist is treating you well!

on August 19, 2008 07:56 PM
# Jonathan Disher said:

Whoops, I missed that ActiveMQ was already in your list. Anyway, I look forward to seeing conclusions on this.

on August 19, 2008 07:57 PM
# alexis said:

Thanks to everyone saying positive things about RabbitMQ. I think this blog post, and the comments beneath it, implicitly raise an important question. Instead of "what open source message queueing systems exist", one could ask "what does the community want from an open source message queueing system?".

Jeremy says he wants something "fast, stable, and have some good replication (think multi-colo) and fault tolerance built-in" so that processes can communicate; and he says "ideally, I'm looking for a system that allows for different message priorities.. [and timeouts]".

What we'd like to know, in order to make RabbitMQ better, is what else do people want? And: how important is each feature?

Cheers,

alexis

on August 20, 2008 05:44 AM
# Justin Mason said:

Alexis -- persistence (either to disk or SQL db) is critical in most of my use cases, I've found. It's great to have super-fast queueing, but if I've enqueued 200k jobs in my task queue and the master queue server's hardware acts up and it reboots, I'd need to ensure that those jobs are still in the queue for processing when the machine is back up.

To be honest, I'm quite mystified by the queueing systems that _don't_ have any persistence-backing on their queues...

on August 21, 2008 04:16 AM
# Jeff Sturm said:

We're in the same position of researching message queueing systems. Thanks to this blog, I have some new pointers I hadn't considered.

OpenMQ works great and looks really solid. I couldn't find any perl bindings for it however, which is a major drawback.

Avoid ActiveMQ unless you don't mind silently dropping messages. It adds something called "virtual topics" to JMS which sounds like a nice idea but are a disaster in implementation.

RabbitMQ looks very interesting. I'd love to hear any stories, good or bad.

on August 21, 2008 04:45 PM
# steve said:

beanstalkd looks like a cool option when it matures a bit. one of my biggest concerns is simplicity. i do not want to implement something that is going to take a lot of time to learn and use - beanstalkd, using the memcached, format should be quick and easy to plugin.

i will be watching this thread for updates.

on August 26, 2008 03:35 PM
# Alex said:

alexis: Since you're asking... I keep looking back at RabbitMQ as an option for some of our requirements but what I'd like to see if a real simple and light-weight REST-like interface to get messages in and out of a queue (I'm thinking along the lines of the way Amazon SQS does things). I realise that the way those queues are implemented behind the scenes for persistence, fault-tolerance etc. can get quite complicated but I don't see why getting messages in and out needs to be.

on September 10, 2008 12:38 PM
# adam said:

Red Hat is betting that a fresh, open source messaging system can displace one of corporate. Red Hat is gambling on Advanced Message Queuing Protocol.

on September 19, 2008 06:01 AM
# Scott Splavec said:

What did you end up choosing? I am investigating open source message queuing systems for my company and was curious what you chose and why.

It would be great if you'd post a follow-up that lists the systems for evaluated and why you did/did not choose them.

on April 22, 2009 02:39 PM
# Rachel Blum said:

Slightly veering off topic, I've been looking all over the place for an OSS Batch/Job queueing system, and the pickings are slim...

Is really nobody interested in coarse parallelization on the process level? (Yes, I've seen condor and torque, but they still require jobs tailored to them. No output logging? Really?)

on June 14, 2009 03:30 PM
# Rob Davies said:

Re-ActiveMQ silently dropping messages - the only reference I've ever been able to find for that is with a miss-use of Spring's JmsTemplate - nothing to do with ActiveMQ - which was behaving correctly.

ActiveMQ is fast, reliable and has failover and high availability - which is why its used by so many enterprises for large deployments. It also doesn't loose messages on hard kills to brokers, and has no memory limit on queue sizes.
When you factor in its an Apache project (so no tie to a single developer or company) and has a very active community - its not surprising its the most widely used open source messaging system.

on December 30, 2009 08:26 AM
# Eugene Portnoy said:

Hi Jeremy,

I arrived here via the link from http://wiki.secondlife.com/wiki/Message_Queue_Evaluation_Notes

I'm architecting a pub/sub system for my company and although our requirements are somewhat different from yours, we are evaluating the same candidates.

I'd love to hear your arguments and a conclusion (if any).

Regards

on January 19, 2010 06:49 PM
# Jeff Brown said:

Eugene,

Have you come to any conclusions? What are your requirements? I would like to hear from you about your investigations, as I am also evaluating MQs.

Jeff

on February 19, 2010 09:18 AM
# John said:

JBoss Messaging, and its follow-on HornetQ, are important contenders, if you want transactional guarantees.

If you have a transactional RDBMS that is as available as you care for your messaging to be, the whole problem is not that hard; the ActiveMQs of the world work under this assumption. However, sometimes you want better availability than your DB or DB team can manage. JBoss Messaging and HornetQ purport to provide this.

on June 2, 2010 10:35 AM
Disclaimer: The opinions expressed here are mine and mine alone. My current, past, or previous employers are not responsible for what I write here, the comments left by others, or the photos I may share. If you have questions, please contact me. Also, I am not a journalist or reporter. Don't "pitch" me.

 

Privacy: I do not share or publish the email addresses or IP addresses of anyone posting a comment here without consent. However, I do reserve the right to remove comments that are spammy, off-topic, or otherwise unsuitable based on my comment policy. In a few cases, I may leave spammy comments but remove any URLs they contain.