Redenormalisation: Building a search index, the other way

What? This post is about how the unobtanium search engine handles normalising words so you don't have to worry about weather a document contains the phrase "cat", "cats" or "cat's" (English is pretty tame here, other languages aren't) and simply can search for "cat" content and find all three using a keyword based search index. This blogpost oversimplifies: Building a search index involves a lot of complexity, to outline this one concpet the rest of the index building is oversimplified here. Turning documents into a search index To start with, lets create a bad search index off of some example documents. Too keep it simple I'll spare you the complexity of actual content and reduce everything to a few keywords. NameKeywords A"cat" "jumps" "silly" B"cats" "sleeping" "cute" C"dog" "jumps" "playing" D"cat" "jumping" "funny" In a very simple search index one maps the keywords to the documents: KeywordDocuments "cat"A, D "cats"B "cute"B "dog"C "funny"D "jumps"A, C "jumping"D "playing"C…