Tokens, Embeddings, and Context Windows Explained

By Kokil Thapa | Last reviewed: September 2026

Tokens, embeddings, and context windows explained in plain engineering terms: three ideas that decide whether your AI integration works, how much it costs, and how far it scales. You send text to an API. The provider turns it into tokens, runs it through a model, and sometimes stores embeddings for search. Every bill, timeout, and truncated reply traces back to those mechanics. If you build chat, RAG, or document workflows on Laravel or WordPress, you need this model in your head before you write the first controller.

What are tokens in large language models?

Tokens are the atomic units a language model actually processes. You type English, Nepali, or JSON. The tokenizer splits that string into integer IDs. The model never sees your raw characters during inference—it sees those IDs and their position numbers.

That gap causes most production surprises. A prompt that looks short in your editor can exceed the context window. A log line with base64 can explode in token count. Billing is per token, not per word.

How tokenization works in practice

Modern APIs use subword tokenizers—often byte-pair encoding (BPE) or similar schemes. Common English words may map to one token. Rare words, typos, and non-Latin scripts split into smaller pieces. Nepali Devanagari text typically uses more tokens per visible character than English prose.

On a legal-tech portal I built, a bilingual FAQ page cost nearly double the English-only estimate. The Nepali paragraphs tokenized aggressively. Always measure; never guess from word count.

Rules of thumb for token counting

Use these estimates only for back-of-envelope planning. Confirm with the provider’s tokenizer before you commit to architecture.

• English prose: about 1 token per 0.75 words, or roughly 4 characters.
• Source code and JSON: often 1.2–2× the token count of equivalent prose.
• Devanagari and mixed scripts: plan for higher token-per-character ratios.
• Output tokens are billed separately from input tokens on most APIs.

The OpenAI tokenizer playground and equivalent tools from other vendors let you paste real prompts and see exact counts. That step belongs in every AI rate limits and cost optimization review.

curl https://api.openai.com/v1/chat/completions \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-4o",
    "messages": [{"role": "user", "content": "Summarise this contract clause."}],
    "max_tokens": 500
  }'

/* Response usage block — log this on every call */
{
  "usage": {
    "prompt_tokens": 842,
    "completion_tokens": 187,
    "total_tokens": 1029
  }
}

In Laravel, persist prompt_tokens and completion_tokens per request. Aggregate by user, feature, and day. Without that telemetry, you cannot explain a Rs 15,000 (~USD 112) monthly API bill to a client.

How do embeddings work in AI applications?

Embeddings are fixed-length vectors—lists of floating-point numbers—that represent the semantic meaning of text, images, or other inputs. Similar meanings produce vectors that sit close together in high-dimensional space. Dissimilar meanings sit farther apart.

That geometry powers semantic search, recommendation, clustering, and deduplication. You do not send embeddings to a chat model for reasoning. You store them in a vector database and retrieve relevant chunks before you call the chat model.

From text chunk to searchable vector

A typical RAG pipeline on a production Laravel app looks like this:

1. Split documents into chunks—often 300–800 tokens with overlap.
2. Call an embedding model API for each chunk.
3. Store vectors plus metadata in PostgreSQL with pgvector, Redis, or a dedicated engine.
4. On user query, embed the question and run nearest-neighbour search.
5. Inject top-k chunks into the chat prompt as context.

I have shipped this pattern on client portals where users search uploaded PDFs. The embedding step is cheap and cacheable. The chat step is expensive and must stay small. See our guide on how to build an embeddings pipeline for ingestion details.

Choosing embedding dimensions and models

Embedding models differ by vector size, language support, and cost. A 1536-dimension model from a major provider is a safe default for English-heavy apps in 2026. Smaller models trade accuracy for speed on edge deployments.

Google’s embedding documentation and OpenAI’s embeddings guide both stress one point: use the same model for ingestion and query. Mixing models breaks similarity math.

/* Laravel job — embed one chunk */
$response = Http::withToken(config('services.openai.key'))
    ->post('https://api.openai.com/v1/embeddings', [
        'model' => 'text-embedding-3-small',
        'input' => $chunkText,
    ]);

$vector = $response->json('data.0.embedding');

DB::table('document_chunks')->insert([
    'document_id' => $documentId,
    'content'     => $chunkText,
    'embedding'   => json_encode($vector),
    'token_count' => $tokenCount,
]);

For Nepali content, validate retrieval quality with real queries—not English-only benchmarks. A Nepali word counter helps estimate chunk sizes before tokenization, but only the API tokenizer gives you billable numbers.

What is a context window and why does it matter?

The context window is the maximum number of tokens a model can accept in a single request. That total includes your system prompt, conversation history, retrieved documents, tool results, and the space reserved for the model’s reply.

Exceed the limit and the API returns an error—or silently truncates input if your client library does not validate first. Both outcomes hurt production apps.

What fills the context window

Think of the window as a fixed bucket. Every byte of prompt competes for the same space.

• System instructions: persona, rules, output format—often 200–800 tokens.
• Few-shot examples: powerful but expensive; each example consumes budget.
• RAG chunks: five chunks at 400 tokens each can eat 2,000 tokens before the user speaks.
• Chat history: grows linearly with every turn unless you prune it.
• Tool JSON: function definitions and API responses add up fast.
• Completion reserve: max_tokens must fit inside the remaining space.

Models advertised with 128K or 200K context windows still charge per token. A larger window removes truncation pain; it does not remove cost. Read our companion piece on long-context LLM strategies and limits for when big windows help versus hurt.

Context window sizes in 2026

Vendor limits change quarterly. Treat published maximums as ceilings, not targets. Latency and cost scale with input length even when the call succeeds.

Anthropic’s model documentation describes context as shared between input and output. OpenAI’s API reference documents the same constraint under max_tokens and model-specific limits. Always read the vendor page for the model string you pass in production.

How do tokens, embeddings, and context windows work together in production?

Tokens, embeddings, and context windows are not three separate features you pick among. They stack. Embeddings shrink what must live inside the context window. Tokens determine whether what remains still fits. The context window caps the entire operation.

On a production Laravel application, I treat this as a pipeline budget problem—not a prompt-writing exercise.

A production budgeting workflow

1. Measure baseline: tokenize your system prompt and tool schemas once at deploy time.
2. Cap RAG: retrieve by similarity, then re-rank and trim to a token ceiling—often 2,000–6,000 tokens.
3. Summarize history: after N turns, compress older messages into a rolling summary stored in Redis.
4. Reserve output: set max_tokens explicitly; never assume unlimited completion space.
5. Log usage: store token counts per feature flag and customer tier.
6. Cache embeddings: hash chunk content; skip re-embedding unchanged documents.

/* Guard before calling the chat API */
$budget = 128_000;
$reservedForOutput = 2_000;
$fixedOverhead = $systemTokens + $toolSchemaTokens;

$availableForContext = $budget - $reservedForOutput - $fixedOverhead;
$selectedChunks = $retriever
    ->search($query, limit: 20)
    ->fitWithinTokens($availableForContext * 0.6);

if ($historyTokens + $selectedChunks->tokenSum() > $availableForContext) {
    $history = $summarizer->compress($history);
}

This mirrors patterns from AI-powered search for Laravel products and standard API development practice: validate inputs server-side, never trust client-side length checks alone.

Common mistakes I see on client projects

Dumping whole PDFs into the prompt. A 40-page contract can exceed 30,000 tokens. Embed and retrieve instead. On document-heavy legal workflows—similar to portals like Mijar Law Associates—RAG is mandatory, not optional.

Ignoring output tokens. A 128K window with 127K input leaves almost no room for the answer. Always subtract max_tokens from your budget math.

Re-embedding on every deploy. Embeddings are deterministic for unchanged text. Version your embedding model in the database schema and migrate deliberately.

Word-count pricing estimates. Stakeholders multiply words by a dollar rate. Developers pay per token. Bridge that gap with logged usage dashboards early.

How do you choose the right approach for your application?

Not every feature needs embeddings. Not every chat needs a 128K window. Match the machinery to the job.

Decision guide by use case

For enterprise application development, define token budgets in the requirements doc the same way you define SLA response times. For content-heavy sites, pair AI features with technical SEO so generated pages still ship with canonical URLs and crawlable structure.

Validate JSON payloads with a JSON formatter during integration testing. Tool-call schemas that balloon silently are a frequent source of context overflow.

Security and compliance notes

Tokens can carry secrets if users paste API keys or PAN numbers into chat boxes. Log token counts, not raw prompts, in production unless you have a data-retention policy and client consent.

Embeddings are not encryption. A vector can leak semantic information about source documents. Treat the vector store with the same access controls as the original files.

External references worth bookmarking: OpenAI’s tokenizer tool, OpenAI embeddings documentation, and Anthropic’s context window guide.

Key Takeaways

• Tokens are billable subword units—count them with the vendor tokenizer before you design prompts or SLAs.
• Embeddings power semantic search; store them once, query cheaply, and never mix embedding model versions.
• The context window is a hard shared budget for input and output—RAG and summarization exist to stay inside it.
• Log prompt_tokens and completion_tokens per request so costs map to features and customers.
• Trim retrieval results and compress chat history before you pay for a larger model tier.
• Match architecture to use case: FAQ bots stay simple; document portals need embeddings plus strict token caps.

People Also Ask

How many tokens is 1,000 words?

For English prose, expect roughly 1,300–1,400 tokens per 1,000 words. Code, tables, and Nepali Unicode text run higher. Paste the actual text into a tokenizer tool rather than using word-count divided by 0.75 in production billing models.

What is the difference between tokens and embeddings?

Tokens are discrete input symbols the language model reads during generation. Embeddings are continuous numeric vectors that represent meaning for similarity search. You convert text to tokens for chat completions; you convert text to embeddings for retrieval and clustering.

What happens when you exceed the context window?

Most APIs return a 400-class error with a message about maximum context length. Some client libraries truncate from the start or end without a loud failure. Always validate total tokens server-side before sending the request.

Do embeddings count toward the context window?

No. Embeddings are produced by a separate model call and stored offline. Only the text chunks you inject into the chat prompt after retrieval consume context window tokens.

Ship AI features with your eyes open

Tokens, embeddings, and context windows explained is the foundation for every sane LLM integration in 2026. Measure tokens, embed documents instead of stuffing prompts, and treat the context window as a budget—not a challenge to fill. That discipline keeps latency predictable and bills explainable to clients in Kathmandu or abroad.

If you want help wiring RAG, usage logging, or guardrails into a Laravel or WordPress platform, see our AI integration and automation service or browse the portfolio for shipped examples. When you are ready to scope a feature with real token math, contact us with a sample prompt and document set—we will tell you what fits before you commit.