Open Source Queueing and Messaging Systems?

RabbitMQ is perfect because it's written in Erlang. Right!?

I think the hardest requirement is: fault tolerance built in. You need to carefully define what that means. For example, some software can actually duplicate each message so that if a provider fails, the message can still be delivered. Others allow two providers to share a disk.

There's also zeromq [http://www.zeromq.org/], which I came upon recently thanks to Delicious.

For the "some messages of less importance" requirement, I think you could technically do this with any of the systems by declaring a different queue. This would also seem to fit better with your example of losing them in a crash: put the low priority stuff on the less fault tolerant system.

Also, can you share any thoughts on why you wouldn't just go with SQS?

http://pi3.informatik.uni-mannheim.de/demaq.html.en is a freely available (though not open sourced) messaging system built on a native XML-Database. Actions on messages are specified in a declarative Xquery like fashion.

I also agree that you need to define what you mean by fault tolerance.

Do you mean that if the plug gets physically pulled on a server that the producers automatically send to different servers and, when that server comes back online, none of the jobs it had in memory are lost?

If so, then I recommend RabbitMQ. Most use cases can be handled by basicPublish which supports priority queues (As shown in the signature here: http://www.rabbitmq.com/api-guide.html#publishing )

I'm recommending RabbitMQ over OpenAMQ and Qpid because they seem to have better/more documentation and an active community. Theoretically though you should be able to swap out RabbitMQ for either OpenAMQ or Qpid and your clients would keep working though. (In practice however there are still small incompatibilities in advanced features but it's close to being transparent)

I'm not sure about Perl bindings for AMQP though. I'm using the excellent Java client RabbitMQ provides but I did notice talk about a perl module for AMQP: http://www.nntp.perl.org/group/perl.modules/2008/06/msg60950.html

All this is if you need the server to keep its messages if the plug is pulled. If that's not a requirement (if an in-memory queue is fine), I'd suggest you look into beanstalkd which has a perl client library: http://xph.us/software/beanstalkd/client.html Most libraries implement dynamic failover I believe (but I'm not 100% on so you should check before believing me!).

Well, I'm partial to XMPP just because it's an IETF RFC and I've played with it and it's readable and extensible and seems generally nice. Since you seem to be looking for Perl, maybe djabberd is a good fit? I seem to remember Brad Fitzpatrick (of LiveJournal fame) also wrote some other Perl queueing software, though I don't remember what it was called. Gearman, or some such? Maybe perlbal was also part of it.

I did a little survey on the subject and OpenMQ from Sun it's a little gem.

https://mq.dev.java.net/

Yeah, ZeroMQ seems to be the latest kid on the block, but I have no experience with it. The site's claims definitely make my mouth water.

Dear jzawodn!

Right now we are working under Perl Client library for RabbitMQ. I cant tell you final date of release but for sure we will do it.

From my previous experience... We was using Spread alot with Perl and this messaging system show himself from best side. It very fast, stable and easy to use. But:)! Spread are...how to say more correct...he is low level. You need to care about almost all things(except multicasting of course:)).

RabbitMQ from this point of view providing much more advanced services. Actually, this advantages cause us start working under Perl Client.

Serg Gulko

SMTP + POP/IMAP.

I'm not into this, but I heard in this podcast:
http://deepfriedbytes.com/podcast/deep-fried-bytes-episode-4-scaling-large-web-sites-with-joe-stump-lead-architect-at-digg/
at digg they're using Gearman:
http://www.danga.com/gearman/
it's not reliable, but looking a little further LiveJournal uses TheSchwartz
http://search.cpan.org/~bradfitz/TheSchwartz-1.07/lib/TheSchwartz.pm
with it to make a reliable MQ system.

We tried Gearman too... But for pure communication(data transmission) I'd recomend use Spread.

Hi,

two systems: TheSchwarts (Danga.com or CPAN, but I recommend the SVN version), and SAPO Broker (http://softwarelivre.sapo.pt/broker/).

Best regards,

I've used both Spread and SQS, but not with the same requirements. Spread gave me a constant 1% cpu utilization, but it is supposed to have guaranteed message delivery. I did notice that some of the Perl clients I wrote to read the Spread stream would flake out sometimes under very high load. I'm sure that was my fault, though.

Regardless of what you go with, please be sure to write a follow-up and share with us what works best for you.

Take a look at JBoss MQ (or whatever they are now calling it). I used it on a prior project that is now pulling over a billion transactions/month. Overall, I found it to scale quite well.

So, having evaluated those toolkits what didn't you like about them? The list of issues would be fascinating.

Personally I've been playing with RabbitMQ and am pretty happy with it. Stayed away from ActiveMQ mostly based on hearsay and horror stories.

check out http://code.google.com/p/sparrow/

you can use any of the perl memcached client libraries to access it.

OpenMQ has quite sparked my interest personally as something for use in OSS, but how does it compare to the commercial {& expensive} package BEA Message Queue? or its commercial package Java System Message Queue? I've used BEA quite a bit in a manufacturing setting, and it has never failed me { /yet/ crossing fingers }. BEA definitely provides a level of service for rock solid messaging in our closed source software systems.

Being a avid supported of OSS, how can we deploy something like Open Message Queue. Being a young software engineer, I'm trying to learn the in's and out's of the plethora of licensing models out there.

We've definitely tried to roll our own "Simple" Messaging Subsystem in our group, and never really ended up with what we needed, or something flexible enough to use again. It always ended up going in the direction of something that resembled a depraved version of BEA's little brother which was much wasted effort.

But not having the correct mindset from management, nor budget allocation puts us in quite the predicament. So I guess I'll just have to pick my poison from what's out there that fits my situation.

http://www.computer.org/portal/site/dsonline/menuitem.9ed3d9924aeb0dcd82ccc6716bbe36ec/index.jsp?&pName=dso_level1&path=dsonline/topics/middleware&file=projects_MOM.xml&xsl=article.xsl&;jsessionid=LqqTjmvSygDp5kspdlv027zSB671S15ZnQdykLl2CkzwmGQ3Q2bQ!-2139236801 is a decent list of some others that are out there...

I may have to play around with RabbitMQ myself here in the next few days to see what it can provide.

Hi Jeremy --

Gearman is nice, fast, perlish and reliable -- we are now using it on the SpamAssassin spamtraps with great results. It's by Brad Fitzpatrick, his stuff is always solid and good perl. It also supports "high priority" messages which skip the queue. However, it doesn't persist messages to disk :(

I understand TheSchwartz is what LJ use to add a more-reliable fault-tolerance layer, but it trades off speed for fault-tolerance. I haven't tried it.

Most of the other new queueing systems are fast, but seem to be lightweight and non-persistent. :(

I have plans to use Gearman soon in another project, with persistence -- I'll add a backup layer of SQL-hosted persistence, myself, to allow recovery of the Gearman queues if the gearmand winds up dying/rebooting etc.

(hey btw I notice that my "via:jzawodny" tag on del.icio.us is now the top hit for "jzawodny" ;)

What was the problem with ActiveMQ? Without cross-checking each of your requirements, I think it would work.

Twitter wrote:

http://rubyforge.org/projects/starling/

"a light-weight persistent queue server that speaks the MemCache protocol. It was built to drive Twitter's backend, and is in production across Twitter's cluster."

Hello,

The following may prove interesting

"JLog is short for "journaled log" and this package is really an API and implementation that is libjlog. What is libjlog? libjlog is a pure C, very simple durable message queue with multiple subscribers and publishers (both thread and multi-process safe)" ...

I have never used it but it seems to have good perl bindings

https://labs.omniti.com/trac/jlog

I was just looking at this a couple of weeks ago - it looks like we have ActiveMQ in dist here at Y!, I'm not aware of who's using it. I personally am looking at using it in my property, it's open source, and appears to have good php and perl bindings, in addition to the usual JMS, etc.

http://activemq.apache.org/

Good luck, if you find anything better I'd love to hear about it. Hope Craigslist is treating you well!

Whoops, I missed that ActiveMQ was already in your list. Anyway, I look forward to seeing conclusions on this.

Thanks to everyone saying positive things about RabbitMQ. I think this blog post, and the comments beneath it, implicitly raise an important question. Instead of "what open source message queueing systems exist", one could ask "what does the community want from an open source message queueing system?".

Jeremy says he wants something "fast, stable, and have some good replication (think multi-colo) and fault tolerance built-in" so that processes can communicate; and he says "ideally, I'm looking for a system that allows for different message priorities.. [and timeouts]".

What we'd like to know, in order to make RabbitMQ better, is what else do people want? And: how important is each feature?

Cheers,

alexis

Alexis -- persistence (either to disk or SQL db) is critical in most of my use cases, I've found. It's great to have super-fast queueing, but if I've enqueued 200k jobs in my task queue and the master queue server's hardware acts up and it reboots, I'd need to ensure that those jobs are still in the queue for processing when the machine is back up.

To be honest, I'm quite mystified by the queueing systems that _don't_ have any persistence-backing on their queues...

We're in the same position of researching message queueing systems. Thanks to this blog, I have some new pointers I hadn't considered.

OpenMQ works great and looks really solid. I couldn't find any perl bindings for it however, which is a major drawback.

Avoid ActiveMQ unless you don't mind silently dropping messages. It adds something called "virtual topics" to JMS which sounds like a nice idea but are a disaster in implementation.

RabbitMQ looks very interesting. I'd love to hear any stories, good or bad.

beanstalkd looks like a cool option when it matures a bit. one of my biggest concerns is simplicity. i do not want to implement something that is going to take a lot of time to learn and use - beanstalkd, using the memcached, format should be quick and easy to plugin.

i will be watching this thread for updates.

Some possibilities...

1. http://www.zeroc.com/ice.html
2. http://www.petercooper.co.uk/archives/001236.html
3. http://isectd.sourceforge.net/
4. http://www.xmlblaster.org/

alexis: Since you're asking... I keep looking back at RabbitMQ as an option for some of our requirements but what I'd like to see if a real simple and light-weight REST-like interface to get messages in and out of a queue (I'm thinking along the lines of the way Amazon SQS does things). I realise that the way those queues are implemented behind the scenes for persistence, fault-tolerance etc. can get quite complicated but I don't see why getting messages in and out needs to be.

Red Hat is betting that a fresh, open source messaging system can displace one of corporate. Red Hat is gambling on Advanced Message Queuing Protocol.

What did you end up choosing? I am investigating open source message queuing systems for my company and was curious what you chose and why.

It would be great if you'd post a follow-up that lists the systems for evaluated and why you did/did not choose them.

Slightly veering off topic, I've been looking all over the place for an OSS Batch/Job queueing system, and the pickings are slim...

Is really nobody interested in coarse parallelization on the process level? (Yes, I've seen condor and torque, but they still require jobs tailored to them. No output logging? Really?)

Re-ActiveMQ silently dropping messages - the only reference I've ever been able to find for that is with a miss-use of Spring's JmsTemplate - nothing to do with ActiveMQ - which was behaving correctly.

ActiveMQ is fast, reliable and has failover and high availability - which is why its used by so many enterprises for large deployments. It also doesn't loose messages on hard kills to brokers, and has no memory limit on queue sizes.
When you factor in its an Apache project (so no tie to a single developer or company) and has a very active community - its not surprising its the most widely used open source messaging system.

Hi Jeremy,

I arrived here via the link from http://wiki.secondlife.com/wiki/Message_Queue_Evaluation_Notes

I'm architecting a pub/sub system for my company and although our requirements are somewhat different from yours, we are evaluating the same candidates.

I'd love to hear your arguments and a conclusion (if any).

Regards

Eugene,

Have you come to any conclusions? What are your requirements? I would like to hear from you about your investigations, as I am also evaluating MQs.

Jeff

JBoss Messaging, and its follow-on HornetQ, are important contenders, if you want transactional guarantees.

If you have a transactional RDBMS that is as available as you care for your messaging to be, the whole problem is not that hard; the ActiveMQs of the world work under this assumption. However, sometimes you want better availability than your DB or DB team can manage. JBoss Messaging and HornetQ purport to provide this.