redjules/End-to-End-Data-Engineering-Project-with-Databricks-Synapse-and-PowerBI ? reverse-engineered prompt

Reverse engineered prompt

Build me an end to end Azure data pipeline for earthquake data, using the USGS earthquake API as the source. I want it to pull new data daily, land the raw data in a bronze layer, clean and normalize it into silver, then create a gold layer with useful aggregates and a country code lookup so it’s ready for analysis.

Please set it up so Databricks handles the notebook processing, Azure Data Factory runs the notebooks in order, and Synapse can query the final Parquet files from storage. Keep the storage organized with separate bronze, silver, and gold containers, and make the workflow feel like a real production setup, with parameters for start and end dates and sensible handling for missing data and duplicates.

If anything needs current setup details from Azure docs, look them up online and use the latest guidance. Add a simple way to visualize or query the results at the end, ideally with Power BI friendly outputs.

Are you gonna build this?

make sure you review the code using coderabbit

Try freeSponsored — opens CodeRabbit in a new tab