Spirizeon/shovel — reverse-engineered prompt
Reverse engineered prompt
Build me a simple command line web crawler in Go that starts from a URL and crawls linked pages really fast without using a lot of memory. I want it to take a starting URL and maybe a few basic options like how many workers to run and how deep to crawl, then print useful progress in the terminal and give me a clean final summary of what it found.
Please make it use goroutines well, avoid crawling the same page more than once, and handle bad links, timeouts, and other normal web errors without crashing. It should feel solid on bigger sites, not just tiny demos, so be thoughtful about queues, concurrency, and memory use.
Keep the code straightforward and easy to read, with sensible defaults and a quick example of how to run it. If anything is unclear, use the project description as the guide and look up current Go docs online if you need to.
Want more depth? Deep Reverse