The LogDrive: Flexible Composition Through Abstraction in Shared Logs

In July, several Confluent colleagues and I published The LogDrive: Composable Durability for Cloud-Based Shared Logs. The paper evolves the concepts of Virtual Consensus in Delos by decomposing the Loglet abstraction into AtomicLog and LogDrive, moving composition below sequencing to a new durability abstraction. This post describes these new abstractions in the context of Virtual Consensus, focusing on how they support RAID-like composability for shared logs.The ideas of the paper came out of a research project led by Mahesh Balakrishnan, Gardner Vickers, and Lucas Bradstreet. The project itself was to build a Kafka-on-S3 service to run in Confluent Cloud and the project ultimately became K2 or Kora 2, the next evolution of the Kora engine (which powers Confluent Cloud’s Kafka service). The first product built on K2 was Freight Clusters (Kafka on S3). The research that led to the LogDrive paper originated in Conflux, the scalable metadata service in K2 which acts as sequencer and…