Is Kafka Costing You More To Operate Than It Should?
Tansu is a modern drop-in replacement for Apache Kafka. Without the cost of broker replicated storage for durability.
Tansu brokers are:
- Kafka API compatible (exceptions: transactions and idempotent producer)
- Stateless with instant scaling up or down. No more planning and reassigning partitions to a broker
- Available with PostgreSQL or S3 storage engines
For data durability:
- S3 is designed to exceed 99.999999999% (11 nines)
- PostgreSQL with continuous archiving streaming transaction logs files to an archive.
Stateless brokers are cost effective, with no network replication and duplicate data storage charges.
Stateless brokers do not have the ceremony of Raft or ZooKeeper.
You can have 3 brokers running in separate Availability Zones for resilience:
Each broker is stateless. Brokers can come and go. Without affecting leadership of consumer groups. The leader and In-Sync-Replica is the broker serving your request. No more client broker ping pong.
With Tansu there is no replication between brokers. The data transfer cost between the Availability Zones is a $0 line item. There are $0 in duplicated storage charges.
With stateless brokers, you can run Tansu in a server-less architecture:
Spin up a broker for the duration of a Kafka API request. Then spin down. No more idle brokers.
Tansu requires that the underlying S3 service support conditional requests. While AWS S3 does now support conditional writes, the support is limited to not overwriting an existing object. To have stateless brokers we need to use a compare and set operation, which is not currently available in AWS S3.
Much like the Kafka protocol, the S3 protocol allows vendors to differentiate. Different levels of service while retaining compatibility with the underlying API. You can use minio or tigis, among a number of other vendors supporting conditional put.
Tansu uses object store, providing a multi-cloud API for storage. There is an alternative option to use a DynamoDB-based commit protocol, to provide conditional write support for AWS S3 instead.