AI doesn't read words like we do. It breaks text into tokens — chunks that might be whole words, parts of words, or punctuation. 'ChatGPT' might be 2-3 tokens. This matters because: 1) You're charged per token on paid APIs, 2) There's a maximum tokens the AI can process at once (context window), 3) Longer outputs cost more. Understanding tokens helps you use AI more efficiently.
AdvancedAI & MLKnowledge
What are tokens?
AI doesn't read words like we do. It breaks text into tokens — chunks that might be whole words, parts of words, or punctuation. 'ChatGPT' might be 2-3 tokens. This matters because: 1) You're charged per token on paid APIs, 2) There's a maximum tokens the AI can process at once (context window), 3) Longer outputs cost more. Understanding tokens helps you use AI more efficiently.
token
Want more like this?
WeeBytes delivers 25 cards like this every day — personalised to your interests.
Start learning for free