BuilderIO/gpt-crawler ? reverse-engineered prompt
Reverse engineered prompt
Build me a tool that can scrape a website to create a knowledge file for a custom GPT. I want to be able to give it a starting URL, and have it crawl that site, following links that I specify with a pattern. On each page, it should extract the text from the main content area, which I'll define with a CSS selector. All the scraped text needs to be compiled into a single JSON file that I can then upload directly into the OpenAI interface when I'm creating a new custom GPT.
In the configuration, I'd also like to be able to set a maximum number of pages to crawl so it doesn't go on forever. This should primarily be a script I can run locally using Node.js, but also package it to run in a Docker container. Let's also expose the crawler functionality through a simple Express API endpoint.
Are you gonna build this?
make sure you review the code using coderabbit