Frontend Beauty
ESC

Type to search the knowledge base.

Tokenizer Gotchas for App Builders

Tokenizers split numbers, spaces, and code in ways that break length checks and cheap string hacks. Count with a tokenizer, not with words.

intermediate7 min read
  • tokens
  • tokenizer
  • cost

A 4-digit invoice number can be several tokens. Your “stay under 100 words” validator lied, the request still blew the budget, and the completion got truncated. A tokenizer does not split on whitespace. It splits the way the model was trained to split: subwords, punctuation glued to letters, leading spaces that are their own story.

If you cap input with value.trim().split(/\s+/).length or string.length, you are measuring a different language than the API bills. Count with the model’s tokenizer (or the vendor’s count-tokens endpoint). Guessing from English averages is how cost alerts fire on a Tuesday.

Key takeaways

  • Word count and string.length are the wrong meters for a context window.
  • Numbers, code, CJK, and leading spaces tokenize unlike prose. Do not memorize a multiplier.
  • Each model family has its own vocabulary. Counts are not portable across vendors.
  • Validate and budget with a real encoder, then add headroom for message framing.

string.length is the wrong meter

JavaScript strings are UTF-16 code units. Tokens are pieces of a model-specific vocabulary. Those two numbers barely correlate once you leave blog-post English.

function wordCount(text: string): number {
  const parts = text.trim().split(/\s+/);
  return parts[0] === '' ? 0 : parts.length;
}

// These checks are the bug, not the safety net.
function allowPrompt(text: string): boolean {
  return wordCount(text) <= 100 && text.length <= 800;
}

async function countTokens(text: string): Promise<number> {
  const res = await fetch('/api/count-tokens', {
    method: 'POST',
    headers: { 'content-type': 'application/json' },
    body: JSON.stringify({ text, model: process.env.LLM_MODEL }),
  });
  if (!res.ok) throw new Error(`count-tokens ${res.status}`);
  const body = (await res.json()) as { tokens: number };
  return body.tokens;
}

async function allowPromptForModel(text: string, budget: number): Promise<boolean> {
  const tokens = await countTokens(text);
  return tokens <= budget;
}

Do not hard-code a token count for "INV-1842" in your app or in this article. Run an encoder if you need a number. The point is the direction of the error: identifiers, JSON, and stack traces usually cost more tokens than the same number of English words.

Server-side counting belongs next to the model id. The browser can still do a cheap character cap as a UX throttle (stop the textarea from pasting a novel). That cap is not the billing cap.

What BPE-style tokenizers actually split

Most LLM tokenizers in production are byte-pair encoding (BPE) or a close cousin (SentencePiece unigram, WordPiece). Hugging Face’s tokenizer summary is the primary explainer. The training process starts with bytes or characters and repeatedly merges the most common pairs into longer tokens.

Consequences you will hit in app code:

Input flavor What tends to happen Why your validator lies
English prose Many whole words are one token This is the case tutorials assume
18422, 2026-11-07 Digits often split into 1-3 digit pieces “One invoice id” is not one token
userId, getById CamelCase and identifiers fragment Code is denser than it looks
Leading space " Hello" Space+word can be a different token than "Hello" Concatenating strings changes counts
CJK text Closer to one character per token A “short” label can dominate the budget
JSON / HTML Braces, quotes, entities, repeated keys Structure is punctuation, and punctuation is tokens
Emoji / rare Unicode Several tokens per grapheme, or a byte fallback string.length and [...str].length still disagree

BPE does not respect JSON syntax. Pretty-printed payloads with newlines and indentation cost more than minified ones. Logging the prettified tool result back into the next turn is a quiet tax.

HTML entities and markdown fences are tokens too. A retrieved page that still has <nav> chrome is not “free context.”

Code, CJK, and the space you did not see

Frontend apps send TypeScript. Tokenizers love to break on punctuation ((, :, =>) and to treat const as a frequent merge while your local helper name stays split. A 80-line component can rival a few pages of docs.

CJK (Chinese, Japanese, Korean) and other non-space-delimited scripts do not get a “word” discount. If your UI copies a Japanese error message into the prompt, budget like you would for a long identifier soup, not like English.

Leading spaces are the gotcha that shows up in log diffs. Many GPT-family tokenizers were trained so that " world" (space included) is a common token, while "world" at the start of a string is another. Building a prompt with parts.join('') vs parts.join(' ') can change the count and the completion. Prefer an explicit template with known separators, then count the final string.

const BAD = ['Status:', status].join(''); // "Status:paid" vs "Status: paid"
const OK = `Status: ${status}`;

async function logPromptSize(label: string, prompt: string) {
  const tokens = await countTokens(prompt);
  console.info(JSON.stringify({ label, chars: prompt.length, tokens }));
}

Log both chars and tokens in traces. When a cost spike lands, you want to see “CJK body” or “minified JSON became pretty,” not a mystery.

For Unicode itself, remember JS length traps: '👍'.length === 2. That is UTF-16, still not tokens. If you already juggle encoding in HTML entities, treat tokenizers as one more encoding, not as “words.”

Validators that lie

maxLength={500} on a textarea. Fine as UX. Insufficient as a quota. Pair it with a server count against the real model.

words * 1.3. A rumor. It undercounts code and CJK, overcounts nothing you can bank on. Do not put it in a billing dashboard.

Counting only the user string. Message roles, tool schemas, and multipart image tokens sit outside that string. See tokens and context windows.

Sharing counts across models. cl100k_base is not Gemini’s tokenizer is not Claude’s. A cache of “this chunk is 412 tokens” is invalid after a model swap unless you key it by tokenizer id.

Truncating with slice(0, 4000) characters to “fit 1k tokens.” You will cut a multibyte character or a JSON key in half, then send a broken last chunk. Truncate on token boundaries with the encoder’s encode + decode round-trip.

function truncateToTokenBudget(
  encode: (text: string) => number[],
  decode: (ids: number[]) => string,
  text: string,
  budget: number,
): string {
  const ids = encode(text);
  if (ids.length <= budget) return text;
  return decode(ids.slice(0, budget));
}

That still can split a word. For RAG chunks, split on headings first, then apply a token budget. Character windows are how you bisect a table.

Count with the model’s tokenizer

Practical options:

  1. Vendor count API (Anthropic token counting, OpenAI token estimates in tooling). Best when you already pay for that model.
  2. Matching local encoder (tiktoken for many OpenAI models, Hugging Face tokenizers for open models). Keep the encoding name next to the model name in config.
  3. Fail closed if the counter is down: refuse the request or skip optional context, do not fall back to word count and hope.

Use the count for packing, caching eligibility, and user-facing “this paste is too large.” Do not use it as a price quote. Prices change. Relative cost (more tokens, more bill; cached input cheaper than uncached) is enough. LLM cost optimization and prompt caching assume you can see token fields on every trace.

Out loud, in an interview

“A tokenizer is a model-specific split of text into vocabulary ids. Word count and string.length do not match that split, especially for numbers, code, CJK, and leading spaces. I would count with the encoder that belongs to the model, truncate on token ids, and key any stored counts by tokenizer id. I would never publish a made-up token count for a sample string.”

Further reading (primary sources)

Related guides