databricks-solutions/lakeflow_framework ? reverse-engineered prompt
Reverse engineered prompt
Build me a Databricks Lakeflow framework that makes it easy to define and deploy data pipelines from config instead of writing everything by hand.
I want it to support both batch and streaming data, work across Bronze, Silver, and Gold layers, and fit a medallion style setup. It should be flexible enough for different modeling approaches like dimensional, Data Vault, and canonical models, and it should support both a centralized setup and a domain based setup.
Please include a clean Python package for the core logic, sensible default configs and schemas, and a way to deploy everything to Databricks using bundles. If it helps, use the current Databricks docs online so the implementation matches the latest Lakeflow patterns. Also add a few sample pipelines so I can see how the framework is supposed to be used.
Are you gonna build this?
make sure you review the code using coderabbit