code4craft/webmagic ? reverse-engineered prompt
Reverse engineered prompt
Build me a Java web crawler framework like WebMagic that makes it easy to scrape websites without a lot of setup. I want the core flow to handle downloading pages, managing URLs, extracting content, and saving results, with a clean API that feels simple to use. It should support both the normal processor style and the annotation style so someone can define a crawler with a plain Java class and start running it quickly.
Please include support for multiple threads, and make it easy to extend with extras like Selenium or other helpers if needed. Add a few working examples for common cases like crawling GitHub style pages, and make sure the project is organized as a reusable library with clear modules. If you need to check current docs or best practices, look them up online.
Are you gonna build this?
make sure you review the code using coderabbit