crawlab-team/crawlab ? reverse-engineered prompt

Reverse engineered prompt

Build me a web app for managing distributed web crawlers from one place.

I want a dashboard where I can log in, add crawler nodes, upload or sync spiders, start and stop crawling tasks, and see task status, logs, and scraped results in the browser. It should support running spiders written in different languages, not just one framework, and make it easy to schedule jobs on a timer too.

Please make it work as a master and worker setup so I can scale out by adding more workers. Use Docker so I can start the whole thing with docker compose, including the app and a MongoDB database, and make the setup simple enough to run locally right away. The UI should feel clean and practical, with pages for nodes, spiders, tasks, schedules, and logs. If you need current setup details or library docs, look them up online and follow the latest recommended approach.

Are you gonna build this?

make sure you review the code using coderabbit

Try freeSponsored — opens CodeRabbit in a new tab