Tokens · Basics
How many tokens is my prompt?
There is no universal “characters divided by four” answer that holds across models and languages. For planning, combine rough heuristics with tokenizer tools or logged usage fields from your provider.
This page explains how to think about prompt tokens, when heuristics fail, and how to connect token counts to dollars using the calculator below.
Token calculation explanation
Tokenizers split text into subword pieces. Short common words may be one token; rare words may become several. Punctuation and whitespace also consume tokens.
Words-to-token examples (indicative)
English prose often lands near three to four characters per token, but JSON, code, and repetitive templates behave differently. Asian languages may produce different ratios than Latin scripts.
Illustrative patterns
- “Hello world” may be two to four tokens depending on model.
- A hundred-line JSON blob may explode versus natural language with the same character count.
Prompt optimization tips
Remove duplicate instructions, collapse whitespace, and prefer bullet constraints over prose when possible.
Token reduction techniques
Summarize stable reference material offline, deduplicate system prompts across services, and avoid sending full chat history when summaries suffice.
Context window explanation
Context limits cap how many tokens a model can consider at once. Approaching the limit increases latency and cost without guaranteeing better answers—curate what you include.
Real pricing examples
Once you estimate tokens, multiply prompt tokens by input price and completion tokens by output price, scaled to per-thousand rates in config/models.php.
Sample prompts by size
| Scenario | Prompt tokens | Output tokens | Model (est.) | Cost / request |
|---|---|---|---|---|
| Short support reply scaffold | 180 | 90 | GPT-4o mini | $0.0001 |
| RAG bundle (5 chunks) | 4200 | 200 | GPT-4o | $0.0125 |
| Code file + instructions | 6500 | 400 | DeepSeek Coder | $0.0010 |
Figures use rates from config/models.php; confirm against your provider before billing decisions.
FAQ: Counting prompt tokens
Short answers mirror the structured data on this page for search engines and readers.
- Should I trust browser heuristics for billing?
- Only for planning—use provider tokenizers or billing logs for invoices.
- Why do two models count differently?
- They use different tokenizers; do not assume parity when switching vendors.
- Do hidden template tokens count?
- Yes—everything sent on the wire counts, including tool schemas.
- How do emojis affect tokens?
- They often consume more tokens than plain ASCII text—test with samples.