projectdiscovery/katana ? reverse-engineered prompt
Reverse engineered prompt
Build me a fast command line web crawler in Go that I can use in automation scripts.
I want to give it one URL, a list of URLs, or piped input, then have it crawl pages and print discovered URLs and useful details to the terminal, a file, or JSON. It should let me control crawl depth, timeouts, retries, headers, proxies, and whether it stays inside the same domain or follows custom scope rules. Include filters so I can exclude private IPs, CDNs, or regex matches.
It should support normal crawling plus an optional headless browser mode for sites that need JavaScript. In headless mode it should be able to find links, JavaScript endpoints, forms, inputs, and XHR requests. Add optional automatic form filling and technology detection.
Please include clear help text, sensible defaults, config file support, error logging, Docker support, and examples in the README. Look up current docs online if you need to.
Are you gonna build this?
make sure you review the code using coderabbit