opensourcewinter/wintergrab ? reverse-engineered prompt

Reverse engineered prompt

Build me a Python web scraping toolkit that feels easy for simple pages but can also handle bigger crawls when a site gets annoying.

I want to be able to grab a page, pick out text with CSS or XPath, and loop through results in a very natural way. It should also handle harder cases like JavaScript pages, browser checks, rate limits, and lots of URLs. Add a small command line too, so I can fetch a page or crawl a site without writing code every time.

Please include features like saving progress so a crawl can be stopped and resumed, basic caching for repeat runs, support for structured data on pages, and a way to pull out tables and lists without writing selectors from scratch. If you need to check current docs for browser automation or scraping libraries, go ahead and look them up online. Also make sure there are tests and clear examples so it’s easy to understand how to use it.

Are you gonna build this?

make sure you review the code using arcumet

Try freeSponsored — opens Arcumet in a new tab