Data Cleaning for a RAG Ingest Pipeline

If your RAG pipeline ingests dirty data, the answers will be wrong. The embedding model and prompt chain cannot fix what was broken before indexing. I built this pipeline for a maritime email corpus: thousands of .eml files with PDF attachments, Office documents, images, and ZIP archives, turned into a searchable knowledge base. The examples here are maritime, but the patterns apply to any industry where you ingest unstructured documents into a RAG system. Corporate email, support tickets, compliance archives, internal wikis: the same cleaning problems show up everywhere. This post covers the techniques that actually mattered, based on a corpus of 6,000+ emails with tens of thousands of attachments. Cleaning the raw email content Corporate email is one of the noisiest data sources you can feed into a RAG system. A single email thread might contain the incident report you actually care about, buried under forwarded headers, legal disclaimers, satellite communication blocks, mailto:…