How Large Language Models Work
Type any sentence and watch it get broken into tokens. Then simulate how an LLM predicts the next word in a sequence.
Token Visualiser
Type or paste any text below. The visualiser will approximate how an LLM tokeniser breaks it into tokens — the fundamental units the model processes.
Click "Tokenise" to see the breakdown
Why this matters: Every token costs money and uses up your context window. A 1,000-word document might be 1,300 tokens. Understanding tokenisation helps you write more efficient prompts and estimate costs.
Next-Token Predictor
See how an LLM decides what word comes next. Choose a sentence starter and watch the probability distribution — then pick the next token yourself to build a sentence.
Key insight: A real LLM has hundreds of billions of parameters and considers ~100,000 possible tokens at each step. This simplified version shows the principle: pick the most likely next word, then repeat. That's the entire mechanism.
Chinese and tokens 中文与词元. Models trained heavily on Chinese, such as DeepSeek and Qwen, include many Chinese characters and common words in their vocabularies. DeepSeek's own documentation gives a rule of thumb: about 0.3 tokens per English character and about 0.6 tokens per Chinese character. A Chinese sentence uses far fewer characters than its English translation, so the same idea often costs a similar number of tokens in either language. Try it: type a sentence in Chinese into the visualiser above. It shows each character as one token, a simple approximation of what real models do.