unclecode/crawl4ai ? reverse-engineered prompt

I want to build a Python tool that can crawl and scrape websites, specifically designed to prepare content for LLMs. The main function is to take a URL, fetch the page, and convert its main content into clean, LLM-ready Markdown. It should strip out all the useless stuff like navigation bars, ads, and footers, leaving just the core text, tables, and code blocks for my RAG pipeline.

It should be able to do a deep crawl, following links to scrape multiple pages on a site. I'd like to use it in two ways: as a simple command-line tool for quick scrapes, and also as an async Python library I can integrate into other projects. A key feature would be the ability to use an LLM to extract specific, structured data—like asking it to pull all product names and prices from a page. And it should be robust enough to handle anti-bot measures and modern JavaScript-heavy sites.