marcelroed/gigatoken ? reverse-engineered prompt

Build me a super fast text tokenizer for language model training in Rust with a simple Python API.

I want it to work like a drop in replacement for HuggingFace tokenizers and tiktoken, so I can wrap an existing tokenizer and use the same encode functions, but I also want a faster mode where it can read text files directly and tokenize huge datasets as quickly as possible. It should support common model tokenizers like GPT 2, Llama, Qwen, Phi, Gemma, and similar ones, and it should be easy to point it at either a model name or an existing tokenizer object.

Make sure it can tokenize batches of text and whole files, and that it feels practical for real training data, not just toy examples. If you need to check current tokenizer docs online, do that. I also want benchmarks and a couple of simple examples so I can see how to use it right away.