huggingface/tokenizers ? reverse-engineered prompt

Build me a fast tokenizer library for natural language text that can train its own vocabulary and then split text into tokens for models like BPE, WordPiece, and Unigram.

I want it to be really quick, easy to use, and good for both research and production. It should handle the usual text prep work too, like normalization, truncation, padding, and adding special tokens. It would be great if it also keeps track of how normalized text lines up with the original sentence, so I can see which part of the input each token came from.

Please include simple examples for training on text files and encoding a string, and make sure it works nicely from Rust, with bindings for Python and Node too if that fits the project. If you need to check the latest docs or examples online, go ahead and do that.