And I kept looking at that dictionary and thinking: nobody is allowed to just use that for compression? TokPress is a pure-Python lossless compressor that tokenizes input with OpenAI's o200k_base ...

A tokenizer is in charge of preparing the inputs for a model. The library contains tokenizers for all the models. Most of the tokenizers are available in two flavors: a full python implementation and ...