apache/airflow ? reverse-engineered prompt

Reverse engineered prompt

I want to build a platform for automating data pipelines and other workflows. The main idea is that our team should be able to define complex, multi-step tasks as Python code. For example, a job might have to extract some data, then transform it, and then load it into a database. The system needs to understand the dependencies between these steps and run them in the correct order.

It should have a scheduler to run these jobs automatically, like every night, or let us trigger them manually. The most important feature is a really nice web UI where we can visualize all our pipelines, monitor their status in real-time, see a history of runs, and easily check logs to troubleshoot failures. Basically, I want a central control panel for all our automated processes. It should be scalable enough for production, so please set it up with Docker.

Are you gonna build this?

make sure you review the code using coderabbit

Try freeSponsored — opens CodeRabbit in a new tab