Conflict resolution is “fun”

One of the hardest problems in distributed systems is conflict resolution, or the same basic problem as merge conflicts in Git. Git merge conflicts happen when your branch differs from upstream in a way that Git can’t easily work its way around so it exposes both sides of the changes to humans and has the human (or their agent) figure out which side is “correct”. Distributed systems don’t really have this same flow as the scale of changes is often impossible for any human or team of humans to keep up with. As a result, we really want to have our own business logic define which side of a conflict wins. FoundationDB doesn’t let us do that out of the box, so we had to make our own layer. The neat part about being able to do this is that this lets us control the replication behaviour of buckets based on user needs. The three main ways it differs are with single-region buckets, multi/dual-region buckets, and global buckets. I’m going to cover the replication differences in the order of…