Building a Medallion Data Pipeline in Azure with Data Factory, Databricks, and Power BI

Why? I have done very little work in Microsoft Azure. I’m very familiar with Google Cloud Platform and the data analytics and AI tools available there, and I’m somewhat familiar with similar AWS offerings. But there’s a technical gap in my Microsoft side. I wanted a quick project to gain some familiarity with the platform. The goal will be to do some simple data transformations and analysis on publicly available data, specifically. To that end, I used two 2023 MEPS files: one that captures person-level demographics, coverage, and expenditures, and another that captures office-based visits at the event level. From there, I shaped the raw data into a simple bronze/silver/gold workflow and used it to analyze cost drivers, utilization trends, and high-cost member concentration. This was all done in a local Jupyter notebook. Now I’m ready to recreate this in Azure. The Plan Here’s the plan. I have the two Excel files, and I’ll need to place them in a storage account in Azure. From there,…