BuilderIO/gpt-crawler ? reverse-engineered prompt

Reverse engineered prompt

Build me a tool that can scrape a website to create a knowledge file for a custom GPT. I want to be able to give it a starting URL, and have it crawl that site, following links that I specify with a pattern. On each page, it should extract the text from the main content area, which I'll define with a CSS selector. All the scraped text needs to be compiled into a single JSON file that I can then upload directly into the OpenAI interface when I'm creating a new custom GPT.

In the configuration, I'd also like to be able to set a maximum number of pages to crawl so it doesn't go on forever. This should primarily be a script I can run locally using Node.js, but also package it to run in a Docker container. Let's also expose the crawler functionality through a simple Express API endpoint.

Are you gonna build this?

make sure you review the code using coderabbit

Try freeSponsored — opens CodeRabbit in a new tab