Link: https://www.warpstream.com/blog/minimizing-s3-api-costs-with-distributed-mmap - WarpStream has built a streaming system with < 1s of latency, without disks. This post walks through the architecture of one of the core components: the distributed segment store - Instead of having brokers, WarpStream has **Agents**, which are completely stateless. Agents are assigned positions in a consistent hash ring. Every Agent is both writing and reading Files that contain potentially several topic-partitions. File IDs are distributed across the agents using the hash ring - Data is written directly to S3 by each agent. Incoming requests are spread over agents. If an agent is assigned the file that the consumer request maps to, they serve it directly. - Else, they use the ring to find the responsible agent, and query it - If the responsible Agent doesn’t have the data cached locally in-memory, it will fetch it from S3 - This system eliminates needless S3 reads for the majority of common cases (set of consumers keeping up with the most recent messages for a topic-partition). And by avoiding inter-AZ networking (see next link) and using S3 as the networking layer for distributing files, they are able to operate at 10-20x cheaper than an equivalent Kafka cluster. - [http://databasearchitects.blogspot.com/2022/04/cloud-network-traffic-within-same.html](http://databasearchitects.blogspot.com/2022/04/cloud-network-traffic-within-same.html) (2022) - All cloud service providers break down their infrastructure into **regions** and **availability zones (AZ)**. - The authors observe that inter-AZ networking costs can be as much as inter-region networking costs. They give AWS as an example: $0.01/GB is charged for both sending _**and**_ receiving traffic between AZs. For comparison, traffic routed across regions is charged at $0.02/GB just to the sender. - They observe (similar to the folks at WarpStream) that if you send data into S3, you can actually circumvent the network cost and save money.