spatie/crawler ? reverse-engineered prompt

Reverse engineered prompt

Build me a PHP package that can crawl a website and collect links in a simple way.

I want to point it at a starting URL, then have it visit pages, follow internal links, and let me limit how deep it goes. It should work well for normal sites, but also handle pages that need JavaScript, so if a page is rendered in the browser it can still crawl it. I’d like a callback when each page is crawled so I can log the URL and status code or do my own processing.

It should also support crawling multiple pages at the same time so it’s reasonably fast, and I want a way to stop the crawl early if some outside condition changes. For testing, please include a fake mode so I can simulate pages without making real network requests.

Make it feel clean and easy to use from PHP, with a nice fluent API and good docs or examples. If you need to check current library docs online, go ahead.

Are you gonna build this?

make sure you review the code using coderabbit

Try freeSponsored — opens CodeRabbit in a new tab