VIDA-NYU/ache ? reverse-engineered prompt
Reverse engineered prompt
Build me a focused web crawler called ACHE that can start from a list of seed URLs and keep finding pages that match a specific topic or domain. I want it to be able to use simple rules like keywords or regular expressions, and also support smarter page classification so it can learn which links are worth following and which ones are not.
It should save crawled pages in a few useful ways, like normal files, JSON, and a search index, and it should have a web interface where I can see what is being crawled in real time and search through the pages it has already found. Please include a simple command line way to start and stop a crawl, plus a REST API for monitoring.
If it makes sense, add support for running in Docker, and make the default setup easy to test with sample config files and seed URLs. Look up current docs online if you need to, but keep it easy to run locally.
Are you gonna build this?
make sure you review the code using coderabbit