marcelroed/gigatoken
Language model tokenization at GB/s
Language:Rust
Total stars: 1987
#rust
#llm, #nlp, #tokenization, #tokenizer