Chapter 2 · Lesson 1 of 4
What Is a Token in AI? Tokens Explained Simply
What Is a Token?
Models do not read whole words the way you do. Text is first chopped into small chunks called tokens. A token might be a full short word like cat, part of a longer word like un + believ + able, a space, or a punctuation mark.
As a rough rule for English, one token is about four characters, or about three quarters of a word. So 100 tokens is roughly 75 words.
Everything a model reads and writes is measured in tokens.
Quick check
The small chunks of text that a model reads and writes are called ______.
Answer it in the game to earn +10 XP.
Ready to play this lesson?
Free · no signup · progress saved in your browser