allenai/dolma ? reverse-engineered prompt

Build me a toolkit for curating large language model training data, with a simple way to process huge collections of documents, tag them with common quality filters, and remove duplicates fast. I want it to work on a laptop for small jobs, but also scale up to a cluster or cloud storage when the dataset is big. Include support for reading from S3 compatible locations, and make it easy to plug in custom taggers if I want to add my own rules later.

Also include a few clear examples or docs so I can see how to run it end to end, from raw text to cleaned output. If anything depends on current libraries or best practices, look up the latest docs online as needed. I’d like the result to feel practical for preparing pretraining data, not just a demo, and to be usable from Python without too much setup.