What is a token in AI?
A token is the unit of text an AI model actually reads and writes — a word, part of a word, or a punctuation mark. Models never see letters or words directly; a tokenizer splits text into tokens first, each gets an ID number, and the model predicts one token at a time. Tokens are also what you're billed for.
Why — the first-principles explanation
A model can't do math on the letter c. It needs numbers. So before anything else happens, a tokenizer chops your text into pieces and assigns each unique piece an ID. "I heard a dog bark loudly at a cat" might become nine tokens, then the sequence of IDs [1, 2, 3, 4, 5, 6, 7, 3, 8] — note that the second "a" reuses ID 3, because it's the same token. The full set of tokens a model knows is its vocabulary. Everything downstream operates on these IDs, never on the text you typed.
So why not just use whole words? Because you'd need a vocabulary entry for every word that exists, and you'd still be helpless the first time someone types a typo, a brand name, or a word in another language. The fix most models use is subword tokenization — GPT models use a method called Byte-Pair Encoding. Common words stay whole; rare ones get split into fragments the model has seen before. That way "unhappiness" can be handled as familiar pieces, and a word invented yesterday is still representable. The tradeoff is real and it cuts both ways: smaller tokens handle anything you throw at them but chew through more of your budget, while larger tokens are cheap to process but break on unfamiliar input. That same sentence tokenized by character is 34 tokens instead of 9.
This is why tokens hit your wallet and your limits. Every model has a context window — a hard cap on tokens covering input and output together. Longer input means less room for output. And AI services bill by the token, usually pricing input and output differently, plus enforcing tokens-per-minute rate limits. When someone says a model has a 200,000-token context, they're telling you the size of its desk, not the size of its brain.
One consequence catches people out: tokens are not distributed fairly across languages. Tokenizers are trained mostly on English, so English text packs efficiently while languages with other scripts often fragment into more tokens for the same meaning. The same paragraph can cost noticeably more in Thai or Hindi than in English. Nobody designed that unfairness on purpose; it fell out of what the tokenizer was trained on.
An example that makes it click
Think of a vending machine that only takes specific coins. You can't feed it a crumpled bill — you have to break your money into coins it recognizes first. The tokenizer is the change machine, and tokens are the coins.
Most common words are a single coin. Rarer ones get broken into two or three. "Cat" costs one coin. Something like "antidisestablishmentarianism" costs a fistful, because the machine has no coin that size and makes change out of smaller pieces it does have. Now here's the part that matters: the machine has a coin slot that holds a fixed number of coins — that's the context window. And you pay per coin. So a long question doesn't just cost more, it also leaves less room for the answer.
Key facts
- A tokenizer splits text into tokens — words, character sets, or combinations of words and punctuation — before the model processes anything; tokenization is the first step in training.
- Each unique token is assigned an ID; "I heard a dog bark loudly at a cat" tokenized by word becomes 9 tokens with IDs [1, 2, 3, 4, 5, 6, 7, 3, 8], the repeated 'a' reusing ID 3.
- The same sentence tokenized by character becomes 34 tokens including spaces — nearly 4x more — showing how tokenization method drives cost and context use.
- GPT models use subword tokenization called Byte-Pair Encoding (BPE); the set of unique tokens a model is trained on is its vocabulary.
- Every model has a maximum context window covering input and output tokens together — longer input leaves less room for output.
- Generative AI services price per request by input and output token counts, often at different rates, and enforce a maximum tokens per minute (TPM) rate limit.
▶ The 60-second explainer (script)
A token is the unit of text an AI actually reads. Not letters. Not words. Tokens. Here's why they exist. A model can't do math on the letter C — it needs numbers. So before anything happens, a tokenizer chops your text into pieces and gives each piece an ID number. Take the sentence: I heard a dog bark loudly at a cat. Split by word, that's nine tokens, and the IDs might be one, two, three, four, five, six, seven, three, eight. Notice the second 'a' reuses ID three — same token, same number. Everything the model does happens on those IDs. It never sees your text. So why not just use whole words? Because you'd need an entry for every word that exists, and you'd still break the first time someone types a typo or a brand name. So most models use subword tokens. GPT uses a method called byte-pair encoding. Common words stay whole. Rare words get split into familiar fragments. That's how it handles a word invented yesterday. Now the part that costs you money. Every model has a context window — a hard cap on tokens, covering input and output together. Longer question, less room for the answer. And you're billed per token, usually at different rates for input and output. One last thing people miss. Tokens aren't fair across languages. Tokenizers are trained mostly on English, so English packs tightly while other scripts fragment. The same paragraph can genuinely cost more in Hindi than in English.
What authoritative sources say
People also ask
How many words is a token?
A common rule of thumb for English is roughly ¾ of a word, or about four characters — so 100 tokens is near 75 words. Treat it as an estimate: the true count depends entirely on the model's tokenizer and your text.
Do spaces and punctuation count?
Yes. Punctuation and whitespace are part of tokenization, and in character-based tokenization spaces are counted individually. This is why formatted text and code often cost more tokens than plain prose of the same length.
Is an AI token the same as a crypto token?
No, and it's an unlucky collision. Crypto 'AI tokens' are tradeable coins from AI-themed blockchain projects. A token here is a chunk of text inside a model — not an asset and not worth anything.
How do I count tokens before sending?
Use the tokenizer that matches your model — OpenAI publishes a visualizer at platform.openai.com/tokenizer, and libraries exist for each major model family. Counts from the wrong tokenizer will be wrong.
Why does my non-English text cost more?
Because tokenizers are trained mostly on English text, so English compresses into fewer tokens. Other scripts fragment into more tokens for the same meaning, which raises both cost and context-window usage.
The same question, asked other ways
- What are tokens in AI?1,600/mo
- What is an AI token?1,300/mo
- What are AI tokens?590/mo