Token

Webisoft AI team Updated 2 min read

A token is the basic unit of text that an AI language model reads and generates, typically a word, part of a word, a number, or a punctuation mark. Before a large language model processes your prompt, the text is split into tokens; the model then predicts its answer one token at a time. In English, a token averages about three-quarters of a word, so 1,000 tokens is roughly 750 words. Common words are usually one token; longer or rarer words split into several (“fine-tuning” might be three).

Tokens matter because everything about a model is measured in them. The context window, how much the model can consider at once, is a token limit. API pricing is per token, usually with separate rates for input (what you send) and output (what the model writes). Speed is quoted in tokens per second. Even odd model behaviors, like miscounting letters in a word, trace back to the model seeing tokens rather than individual characters.

Why it matters at work

For everyday chat use you can ignore tokens. They start to matter the moment your team builds anything on an AI API or processes documents at scale: token counts determine what fits in one request, how much each workflow run costs, and where a long report gets silently truncated. A team summarizing 10,000 customer reviews a month is really buying tokens, and prompt length becomes a line item, trimming a bloated 2,000-token instruction block that rides along on every request is a real cost and latency win.

A work example

An operations lead budgeting an AI document-processing pipeline estimates each 20-page contract at roughly 10,000 tokens of input, multiplies by per-token pricing and monthly contract volume, and gets a defensible cost forecast before writing any code.

Get one practical AI-at-work idea in your inbox each week, subscribe to the AI Work+ newsletter.

FAQ

How many tokens is a page of text? A single-spaced page runs roughly 500 words, which is around 650 to 700 tokens in English. A useful rule of thumb: one token is about three-quarters of a word.

Why do AI providers charge by token instead of by word? Tokens are the units the model actually processes, so they track compute cost directly, and they work consistently across languages, code, and punctuation where word counts get ambiguous.