In a previous blog, we gave an overview of the different messaging protocols available to us (AMQP & JMS) and listed each one’s benefits and issues. In this blog, we intend throwing light on the different messaging solutions available in the market such as Kafka, RabbitMQ, Cloud Messaging solutions such as Amazon SQS and Google Pub Sub, Container built in messaging such as Oracle M)M in Oracle Weblogic and IBM MQ in WebSphere and which should be used for what situation
Apache Kafka is a message broker that was originally developed by LinkedIn before going open source under the Apache umbrella. Kafka’s main focus was speed and efficiency. It can maintain a message throughput rate of 100k+ msgs/sec which is significantly higher than most other alternatives. However, other basic features found in other brokers are eschewed in favor of efficiency. Features like message acknowledgments, routing rules and an admin UI are not available as standard. Even though Kafka is defined as ‘stable’ and ‘production-ready’, deploying Kafka in HA mode is tougher than it is with other providers. It is highly dependent on the Zookeeper module in order to maintain synchronization between multiple Kafka nodes and client apps.
RabbitMQ is an open source message broker software managed by Pivotal. It implements the Advanced Message Queuing Protocol (AMQP) which we described in detail in an earlier blog. Unlike Kafka, RabbitMQ comes with a built-in management dashboard for easy configuration and monitoring of message queues. It also supports requests for message acknowledgements and is easily deployed in HA mode with minimal effort. With a message rate of about 20k+ msgs/sec which is much less than Kafka, it’s sufficient enough for most use cases. Due to it’s platform-independent framework and ease of use, RabbitMQ is a very mature message broker offering with a lot of extended support from the community.
Simple Queue Service (SQS) is Amazon’s cloud-based message queuing system made available as part of Amazon Web Services (AWS). Unlike the other brokers mentioned here, there is barely any setup/deployment required for an application to use SQS. All you need is AWS credentials to be able to use it. Since it is a SaaS (Software-as-a-service), this makes it a much lower-cost option since there is no infrastructure cost. You only pay for what you use. SQS also has the notion of in-flight messages. This means that the message is pulled off the queue but never deleted until the ‘delete‘ command is sent for that message ID. So if you lose your worker mid-processing of the event, that event isn’t lost for good.
One drawback of the SQS implementation is the need for polling of a message queue to determine if new messages have appeared. This is a bit of an issue being as you must now model your application to perform a polling cycle in order to determine if new messages are available. While the SQS costs are reasonable, Amazon places a limit on the message rate which can be increased on request and also charges extra based on the overall message rate.
Google Cloud Pub/Sub
Google’s Cloud Pub/Sub is a fully-managed real-time messaging service that allows you to send and receive messages between applications or server. It is developed specifically to faciliate messaging among applications deployed on the Google Compute Engine (GCE) cloud. It’s basic features are very similar to that of Amazon’s SQS in that there are no infrastructure costs and you pay for what you use. The major difference from SQS is that Google charges based only on volume and not transacation speed i.e. you will not be charged extra for a sudden increase in message rate.
IBM MQ & Oracle WebLogic JMS
Both IBM MQ and Oracle WebLogic JMS are very similar in features to standard JMS. You would choose one over the other based on what framework your app is using i.e. if your app is deployed on IBM Websphere, it will be much easier to integrate it with IBM MQ than any other provider.
Similarly, it would be simpler to integrate an app running on Weblogic with the JMS server that it provides instead of an external provider.
|Category||Apache Kafka||RabbitMQ||Amazon SQS||Google Pub/Sub||Weblogic/ Websphere JMS|
|Message Rate (msgs/sec)||100k+||20k+||Varies
(can be scaled)
|Message Acknowled- gements||No||Yes||Yes||Yes||Yes|
|HA||Yes (using Zookeeper and multiple nodes)||Yes||Yes||Yes||–|
|Protocol||Kafka||AMQP||HTTP REST||HTTP REST,RPC (experimental)||JMS|
|Additional Features||Can be enhanced by integrating with Apache Storm||Several plugins available to add more features||In-flight messages||–||–|
|Cost||High||Medium||Pay-per-use||Pay-per-use||High (license is bundled with package)|
|Supported Frameworks||C/C++, Python, Go,Erlang, .NET, Java,Ruby, Node.js, Perl, PHP||C/C++, Python, Go,Erlang, .NET, Java,Ruby, Node.js, Perl, PHP||C/C++, Python, Go,Erlang, .NET, Java,Ruby, Node.js, Perl, PHP||C/C++, Python, Go,Erlang, .NET, Java,Ruby, Node.js, Perl, PHP||Any JMS-compatible client|
|Use Cases||High ingestion platforms where speed,scale and efficiency is the prime concern||Robust systems where features like acknowled- gements, deeper management are important||Useful when deploying applications on AWS EC2, Elastic Beanstalk to facilitate low-latency communication||Works well with applications deployed on GCE||Usually used as default message provider in their corresponding packages|
- Introduction to Messaging
Messaging is one of the most important aspects of modern programming techniques. Majority of today's systems consist of several modules and external dependencies. If they weren't able to communicate with…
- DiP (Storm,Spark,Flink and Apex) Co-Dev opportunity
Real time data ingestion using Data Ingestion Platform (DiP) which harness the powers of Apache Apex, Apache Flink, Apache Spark and Apache Storm to give real time data ingestion and visualization. DiP…
- Real time data ingestion – Easy and Simple (co-dev opportunity)
This work is based on Xavient co-dev initiative where your engineers can start working with our team to contribute and build your own platform to ingest any kind of data…
- Introduction to Microservices
Traditional development methodologies encourage the ‘monolithic’ approach to application development. Building a single application that does everything required has been the modus operandi for a while. However, with the rise…
- Real Time Data Ingestion (DiP) – Spark Streaming (co-dev opportunity)
This blog is an extension to that and it focuses on integrating Spark Streaming to Data Ingestion Platform for performing real time data ingestion and visualization. The previous blog DiP (Storm Streaming) showed how…
- Real Time Data Ingestion (DiP) – Apache Apex (co-dev opportunity)
Data Ingestion Platform This work is based on Xavient co-dev initiative where your engineers can start working with our team to contribute and build your own platform to ingest any…