See exactly how text splits into tokens.
How to use Tokenizer Playground
Enter your text. Type or paste text into the Text to tokenize box. It starts with a sample sentence you can replace.
Read the counts. Below the box you see the character count, the token count and the vocabulary size.
Inspect the tokens. The Tokens panel shows each token as a colored chip. Hover a chip to see its id. A marker shows a token that begins with a space.
Copy the ids. Press Copy ids to copy the token ids as a comma-separated list.
Limits and privacy
- This is an illustrative byte-level BPE tokenizer with a regex pretokenizer and a small merge list trained on a local sample corpus. The vocabulary has a few hundred entries.
- It is not the real tokenizer of GPT, Claude, Llama or any other specific model, so token counts and ids will differ from those models.
- Use it to see how BPE splits text, not to get exact token counts for billing or context limits.
- Tokenization runs in your browser on every keystroke. Nothing is sent anywhere.
Frequently asked questions
Does this match the GPT or Llama tokenizer?
No. It implements the same kind of byte-level BPE algorithm, but with its own small vocabulary, so boundaries and ids will not match any specific production model.
Can I use it to count tokens for my API bill?
Not reliably. Use the provider's own tokenizer for exact counts. This tool shows how BPE behaves in general.
Why are some words one token and others split up?
BPE merges frequent byte pairs, so common words and affixes collapse into single tokens while rare or unusual text stays fragmented into smaller pieces.
Is my text uploaded?
No. The tokenizer runs entirely in your browser.