gocolly/colly ? reverse-engineered prompt
Reverse engineered prompt
Build me a clean Go web scraping and crawling library that feels easy to use for normal Go developers. I want to be able to create a crawler, visit a starting web page, follow links, and pull structured data from HTML or XML without writing a bunch of boilerplate.
It should handle the annoying parts for me, like cookies, sessions, request delays, per domain limits, caching, character encoding, robots.txt, and running jobs either one at a time or in parallel. Please make it fast, but also give users control so they do not accidentally hammer a site.
Add a simple event style API where users can run code when a request starts, when a response arrives, when an HTML element matches, and when there is an error. Include support for extensions, environment based config, queues or storage for bigger crawls, and clear examples that show link crawling and data extraction.
Please write tests and keep the public API simple and well documented.
Are you gonna build this?
make sure you review the code using coderabbit