In the modern data platform world, companies rely heavily on data-driven decisions. However, the question remains: Are we using the data at the right time? The answer may not always be ‘yes.’ Companies in every industry quickly shift from batch processing to real-time data streams to keep up with modern business requirements. The specific use cases such as fraud detection, contextual marketing triggers, and dynamic pricing rely on leveraging a data feed or real-time data. Based on the current and comprehensive data, organizations get visibility to make informed operational decisions faster.
When deciding on real-time data ingestion solutions, customers look for the following factors:
- Scalability
- Ordering
- Consistency and durability
- Fault tolerance and data guarantees
- Cost effectiveness
AWS MSK Serverless and Confluent Kafka Standard are considered to be the best services for real-time data ingestion. This blog shares insights that will help you make an informed decision on the right fit for your requirements.
About AWS MSK Serverless
Amazon Managed Streaming for Apache Kafka (Amazon MSK) Serverless is a fully managed service that enables you to build and run applications that use Apache Kafka to process streaming data. It automatically provisions and scales capacity while managing the partitions in your topic, so you can stream data without thinking about right-sizing or scaling clusters. It offers a throughput-based pricing model, so you pay only for what you use.
About Confluent Kafka Standard
Confluent Kafka standard provides a truly cloud-native experience, completing Kafka with a holistic set of enterprise-grade features to enhance developer productivity, operate efficiently at scale, and meet your architectural requirements before moving to production. Underpinning the platform is its 99.99% uptime SLA and committer-driven expertise, providing support and services from the team with over one million hours of technical experience with Kafka.
Let us now compare AWS MSK Serverless and Confluent Kafka Standard across different focus areas:
1. Architecture
| Criteria Group | AWS MSK Serverless | Confluent Kafka Standard |
|---|---|---|
| Ease of Setup | Run Apache Kafka without having to manage and scale cluster capacity as a serverless offering | No setup is involved as Confluent Kafka is serverless |
| Access Controls | Amazon MSK uses IAM to check whether the client is an authenticated identity and is authorized to interact with your cluster | Confluent Cloud role-based access control (RBAC) lets you control access to an organization, environment, cluster, or granular Kafka resources based on predefined roles and access permissions |
| Ecosystem integration | Integrates with AWS ecosystems like VPC, Lambda, Glue Schema, and AWS Kinesis Analytics that process streaming data | Offers Pre-built Kafka Connectors, Confluent Hub, Schema Registry and MQTT Proxy |
| Scalability | In serverless clusters, Amazon MSK automatically balances partitions | Automatic resource allocation to your cluster to manage consumer lag as throughput scales up or down with self-balancing clusters |
| Availability | Amazon MSK uses multi-AZ replication for high availability. Data replication is included at no additional cost | Standard clusters are designed for production-ready features with an uptime SLA: 99.95% for Single-Zone and 99.99% for Multi-Zone |
2. Platform Support
| Criteria Group | AWS MSK Serverless | Confluent Kafka Standard |
|---|---|---|
| Offerings | AWS MSK and MSK Serverless are the two offerings provided by AWS for the managed Kafka | Self managed and fully managed are the two offerings provided by Confluent for managed Kafka |
| Connectors | Create custom plugins using MSK connect to move data between source and destination systems | 120+ pre-built connectors for real-time integration between source and destination systems |
| Kafka Upgrades | MSK Serverless automatically upgrades clusters without requiring customers to provide any input | Upgrade policy in confluent Kafka is broadly classified into three types - minor, major and deprecation upgrades listed here |
| Quota | Max ingress throughput : 200 MBPS Max egress throughput : 400 MBPS Ingress per partition : 5 MBPS Egress per partition : 10 MBPS Max partition size : 250 GB |
Max ingress throughput : 250 MBPS Max egress throughput : 750 MBPS Ingress per partition : 5 MBPS Egress per partition : 15 MBPS Max partition size : 250 GB |
| Monitoring | Amazon MSK integrates with Amazon CloudWatch so that you can collect, view and analyze CloudWatch metrics for your Amazon MSK cluster | Confluent Control Center is a web-based tool for managing and monitoring Apache Kafka®. Control Center provides a user interface that provides you with a quick overview of cluster health, observe and control messages, topics, and Schema Registry, and develop and run ksqlDB queries |
| Workloads | Workloads which are AWS native and need deeper integration with other AWS services are a good fit | Workloads which are hybrid type, i.e on-prem and cloud/multi-cloud are suited for Confluent Kafka platform |
3. Data Governance and Security
| Criteria Group | AWS MSK Serverless | Confluent Kafka Standard |
|---|---|---|
| Compliance Certificates | HIPAA eligible, PCI, ISO, SOC 1,2,3,FedRAMP | SOC 1/2/3, ISO 27001 and PCI, GDPR |
| Network Security | MSK provides a strong network security pillar using AWS VPC Peering, AWS Transit Gateway and AWS Private link to provide a secure traffic flow between cross-account and cross-regions | All the Confluent Standard clusters are accessible through secure internet endpoints. All connections to Confluent Cloud are encrypted with TLS and require authentication using API keys, regardless of network configuration |
| Data Encryption | AWS KMS for data in rest and transit encryption. | Bring your own key (BYOK) for at-rest encryption. Data in motion encryption is available |
4. Pricing
| Criteria Group | AWS MSK Serverless | Confluent Kafka Standard |
|---|---|---|
| Connector pricing | MSK connect pricing breakdown | Connectors in Confluent cloud breakdown as per source |
Scenario:
Let’s assume the cluster has 5 topics with 20 partitions each. Your producers write on average 100GB of data daily and your consumers read 200GB of data. You also retain that data for 24 hours to ensure it is available for replay. In the above scenario, you would pay the following for a 31-day month:
Here’s the pricing breakdown for MSK Serverless (for Ohio Region)
| Charges | Usage | Rate(USD) | SubTotals |
|---|---|---|---|
| Cluster/hours | 31 days * 24 hrs/day = 744 cluster-hours | 0.75/cluster-hr | 744 * 0.75 =558 |
| Partition/hours | 31 days * 24 hrs/day * 5 * 20 = 74,400 partition-hours | 0.0015/partition-hr | 74,400 * 0.001875 = 111.60 |
| Data-in | 100 GB x 31 days = 3,100 GB | 0.10/GB-in | 3,100 * 0.10 = 310 |
| Data-out | 200 GB x 31 days = 6,200 GB | 0.05/GB-out | 6,200 * 0.05 = 310 |
| Storage | Average storage used = 100 GB-months | 0.10/GB-month | 100 * 0.10 = 10 |
| Total | 1229.60 USD |
Here’s the pricing breakdown for Confluent Kafka Standard:
| Charges | Usage | Rate(USD) | SubTotals |
|---|---|---|---|
| Cluster/hours | 31 days * 24 hrs/day = 744 cluster-hours | 1.5/cluster-hr | 744 * 0.75 =1116 |
| Partition/hours | First 500 partitions at no additional cost | 0.0015/partition-hr | 0 |
| Data-in | 100 GB x 31 days = 3,100 GB | 0.13/GB-in | 3,100 * 0.13 = 403 |
| Data-out | 200 GB x 31 days = 6,200 GB | 0.06/GB-out | 6,200 * 0.06 = 372 |
| Storage | Average storage used = 100 GB-months | 0.10/GB-month | 100 * 0.10 = 10 |
| Total | 1901 USD |
What do you select?
You should choose AWS MSK when
- You need more seamless and deep integration with other native AWS Services like AWS lambda for MSK event sourcing, AWS Secret Manager for client credentials used for SASL/SCRAM authentication, etc.
- You want to lower costs since the cluster hours are reasonably less.
- You need speed in provisioning Kafka Cluster
- You need secure connectivity to your MSK Cluster and other clients accessing the resource using AWS Private Link, VPC Peering or Transit Gateway.
You should choose Confluent Kafka when
- You have a hybrid and multi-cloud strategy (available on AWS, Azure, GCP), making it native to the public cloud providers.
- You need to centralize Kafka Management Operation and have a quick overview using the Confluent Control Center.
- You have an on-premises deployment with cloud providers (e.g., AWS Outpost including Wavelength, Google’s Anthos) since it's primarily built on top of Kubernetes
- You need a rich Kafka ecosystem like pre-built connectors, governance using stream lineage, and the ability to connect non-java Kafka clients.
AWS MSK Serverless and Confluent Kafka Standard are both efficient, reliable, and among the market's best real-time data ingestion solutions. However, they cater to different uses and processes. Hence, evaluating the use case and selecting the most suitable tool is important.
Quantiphi as Amazon MSK Service Delivery Partner
Quantiphi is a designated service delivery partner that makes it easy for customers to migrate and build data streaming solutions on Amazon MSK to not only take advantage of the rich Amazon MSK integrations with other AWS services and address real-time analytics use cases but also help them realize the cost benefits sooner. To get started, or learn more, get in touch with our experts.
