kwan-0915/AutoCrawler ? reverse-engineered prompt

Reverse engineered prompt

Build me a simple Python app that can automatically crawl websites and save the results into a local data folder.

I want to be able to give it a starting URL and have it visit pages, collect the important content, and store the output in a clean format like JSON or CSV so I can use it later. It should handle basic crawling rules, avoid getting stuck in loops, and keep track of what it has already visited. If possible, make it easy to change things like the starting page, how many pages to crawl, and what data to keep.

Please make it work from one main script, and if you need to look up current best practices or docs online, go ahead. Keep it simple, practical, and easy to run locally.

Are you gonna build this?

make sure you review the code using coderabbit

Try freeSponsored — opens CodeRabbit in a new tab