tokenizers

1

Rust alternative to Hugging Face tokenizers with Oniguruma, ending C regex in the tokenizer hot path

tokenizers is the Remade with Rust pin for this lane: keep the product shape, change the safety floor, and stop shipping the old C or cloud default by accident.

2

Instant benefit

Provides an implementation of today's most used tokenizers, with a focus on performance and versatility.

3

Deploy with GitHub access

1. Confirm you can open https://github.com/Remade-With-Rust/tokenizers.
2. Add the crate to your project (crates.io when published, otherwise git):

   cargo add tokenizers

   Git pin: cargo add --git https://github.com/Remade-With-Rust/tokenizers.git

3. Prove the pin with cargo check before the next agent prompt can swap it for a C or cloud default.
4. Read the builder field note for composition: /learn/tokenizers/.