How to Evaluate LLM Outputs (Evals)

By Kokil Thapa | Last reviewed: September 2026

You cannot trust a demo. A chatbot that nails three prompts in a meeting can fail on real user questions an hour after launch. How to Evaluate LLM Outputs (Evals) is the discipline that turns guesswork into repeatable quality checks. On production Laravel apps where I wire OpenAI or Anthropic APIs into booking flows and document Q&A, evals sit between prompt engineering and LLMOps. This guide walks through datasets, metrics, human review, and CI gates you can run today.

How Do You Evaluate LLM Outputs Before Shipping to Production?

Start with the user journey, not the model card. List every task where an LLM touches business logic: summarising legal intake forms, drafting product descriptions, classifying support tickets, or extracting JSON from unstructured text. Each task needs its own eval suite because accuracy on one prompt type says nothing about another.

An eval case is a tuple: input prompt, optional context documents, expected behaviour, and scoring rules. Store cases in version-controlled JSON or YAML so prompt changes trigger measurable diffs. Tie cases to real failures you have seen in staging or support logs. Synthetic-only datasets miss the long-tail queries that break production apps.

Define pass criteria per task type

Classification tasks need exact label match or F1 against a label set. Extraction tasks need schema validation—parse JSON and assert required keys. Generation tasks need rubric-based scoring because there is rarely one correct paragraph. Write the rubric before you look at model answers. Rubrics written after the fact drift toward whatever the latest model produced.

Capture production-shaped inputs

Redact PII before adding cases to your repo. Follow the same rules you use to protect PII in LLM apps. Store hashed identifiers if you need to trace a case back to a ticket. Never commit API keys, client documents, or live credentials alongside eval fixtures.

What Metrics Should You Use to Evaluate LLM Output Quality?

Pick metrics that map to user pain. A support bot that sounds fluent but cites wrong policy numbers is worse than one that says "I am not sure." Match the metric to the failure mode your stakeholders actually care about.

Combine cheap deterministic checks with slower semantic checks. Regex and JSON Schema catch format regressions in milliseconds. Embedding similarity or LLM-as-judge catches paraphrased wrong answers that still look structurally valid. I run both layers on AI integration projects because either layer alone leaves blind spots.

Deterministic checks first

Validate structured outputs and JSON mode with a schema library. Assert required fields, enum values, and numeric ranges. Use your JSON formatter during manual debugging, but automate validation in CI. Regex works for dates, phone patterns, and citation markers when the format is fixed.

Semantic checks second

Embedding cosine similarity between model output and a reference answer works for short factual responses. It fails when multiple valid phrasings exist. LLM-as-judge—asking a separate model to score against a rubric—handles nuance better but introduces its own bias. Calibrate the judge against human labels on a sample before trusting aggregate scores.

How Do You Build a Golden Dataset to Evaluate LLM Outputs?

A golden dataset is your regression suite for language models. Treat it like application test fixtures. Each row should represent a real failure you never want to see again, plus a handful of happy-path cases to catch overfitting to edge cases only.

1. Collect 30–50 anonymised prompts from logs, support tickets, or staged user sessions.
2. Write reference outputs or rubric criteria with a domain expert—not only with the model.
3. Tag cases by category: format error, hallucination, wrong tone, tool misuse, safety.
4. Commit the dataset beside your application code or in a private eval repo.
5. Re-run the full suite whenever prompts, retrieval chunks, models, or temperature change.

On a legal-tech portal I maintain, eval cases cover Nepali and English intake questions, document checklist accuracy, and refusal behaviour when users ask for specific legal advice. Those cases mirror patterns from Court Marriage In Nepal and similar sites where wrong guidance has real consequences. Generic trivia benchmarks do not substitute for domain cases.

Sample eval case in JSON

{
  "id": "intake-014",
  "task": "document_checklist",
  "input": {
    "user_message": "What do I need for court marriage if one party is foreign?",
    "locale": "en"
  },
  "context_files": ["faq-court-marriage-v3.md"],
  "expected": {
    "must_mention": ["passport", "unmarried certificate", "witness"],
    "must_not_contain": ["guaranteed approval", "exact fee amount"],
    "response_format": "markdown_list"
  },
  "rubric": {
    "faithfulness": "Only facts present in context_files",
    "completeness": "All three document types listed",
    "tone": "Informational, not legal advice"
  }
}

Store retrieval context paths, not full document bodies, when files are large. Your eval runner should load the same chunk pipeline your app uses in production. Evaluating against stale static context while production uses live RAG produces false confidence.

Grow the dataset from incidents

Every production miss becomes a candidate case. When monitoring and guardrails flag a bad answer, export the prompt, retrieved chunks, model version, and final text. Add it within 48 hours while context is fresh. Teams that wait for quarterly reviews accumulate the same failures repeatedly.

How Should You Run LLM Output Evals in CI and Before Deploy?

Evals belong in the same mental bucket as PHPUnit or Pest—not as optional research homework. Wire them into GitLab CI, GitHub Actions, or your existing pipeline beside lint and unit tests. Full suites against paid APIs cost money, so tier them.

Run a fast smoke eval on every pull request: 10–20 critical cases, deterministic checks only, finish under two minutes. Run the full golden dataset nightly or before release tags. Block merge when smoke eval pass rate drops below your agreed floor—commonly 90% for format tasks, lower for open generation with human spot checks.

Minimal PHP eval runner sketch

Laravel apps often call LLMs from a service class. Your eval runner reuses that class with frozen config. Log every inference parameter so you can diff failures across runs.

<?php
/* tests/Eval/LlmEvalRunner.php — simplified pattern */

final class LlmEvalRunner
{
    public function __construct(
        private LlmClient $client,
        private array $cases,
    ) {}

    public function run(): array
    {
        $results = [];

        foreach ($this->cases as $case) {
            $response = $this->client->complete(
                prompt: $case['input']['user_message'],
                context: $case['context_files'] ?? [],
                model: env('EVAL_MODEL', 'gpt-4.1'),
                temperature: 0,
            );

            $results[] = [
                'id' => $case['id'],
                'schema_ok' => JsonSchemaValidator::check($response, $case['expected']),
                'must_mention' => KeywordAssert::allPresent($response, $case['expected']['must_mention']),
                'judge_score' => optional JudgeLlm::score($response, $case['rubric']),
            ];
        }

        return $results;
    }
}

Pin temperature to 0 for eval runs unless you explicitly test sampling variance. Compare models on the same case set when you evaluate cost swaps—see LLM cost optimization for the business side. A cheaper model that loses five points on faithfulness may still be the right trade-off if you document the gap.

Log metadata for every run

Record model name, prompt template hash, retrieval index version, and API latency. Without metadata, a failing nightly run tells you nothing actionable. Structured logs feed the same dashboards you use for testing and optimization on non-AI features.

When Do Human Review and Red Teaming Beat Automated Evals Alone?

Automated evals scale. Humans catch what rubrics miss: subtle policy violations, culturally wrong tone, or answers that sound authoritative while being incomplete. Plan for both.

Sample 5–10% of live traffic for human review when you first launch a feature. Drop to 1–2% once scores stabilise, but never zero for high-risk domains. Pair human review with red teaming LLM applications to probe jailbreaks, prompt injection via retrieved content, and function-calling abuse paths automated cases skip.

LLM-as-judge calibration

When you use a judge model, calibrate it against human labels on at least 50 cases. Measure agreement rate. If the judge disagrees with humans more than 15–20% on critical fields, rewrite the rubric or switch judges. OpenAI documents evaluation patterns in their Evals guide. Anthropic covers similar ground in their test and evaluate overview. Read both—vendor defaults differ, and your app constraints matter more than their examples.

Align with governance expectations

Document who owns the eval suite, how often cases refresh, and what score triggers a rollback. That paperwork supports AI governance basics without slowing delivery. For user-generated content pipelines, cross-check automated scores with AI content moderation rules so evals and live filters do not diverge.

Key Takeaways

• Build a version-controlled golden dataset from real prompts and documented expected behaviour—not from generic trivia benchmarks alone.
• Stack deterministic checks (JSON Schema, regex, exact match) before semantic scoring (embeddings, LLM-as-judge).
• Run smoke evals on every PR and full suites before release; block deploys when aggregate scores drop.
• Calibrate LLM-as-judge against human labels before trusting rubric scores on high-risk tasks.
• Add human sampling and red teaming for customer-facing, legal, or payment-adjacent features.
• Log model ID, prompt hash, and retrieval index version on every eval run so failures are reproducible.

People Also Ask

What is an LLM eval?

An LLM eval is a repeatable test that sends a fixed prompt through your production configuration and scores the response against expected behaviour. Evals catch regressions when you change prompts, swap models, or update retrieval documents. They work like unit tests, but outputs are probabilistic so you score with rules and rubrics instead of a single assert equals.

How many eval cases do you need?

Start with 30–50 cases covering your highest-risk tasks and known failure modes. That is enough to catch most prompt regressions in CI. Grow the set continuously from production incidents. Breadth across task types beats raw volume—500 near-duplicate cases add noise without signal.

Is LLM-as-judge reliable?

It is useful but not infallible. Judge models favour fluent, confident answers—even wrong ones. Always calibrate against human-reviewed samples on your domain. Use judges for ranking and rubric scoring, not as the sole gate for legal, medical, or financial outputs.

How often should you rerun evals?

Run a small smoke set on every pull request that touches prompts, RAG content, or model config. Run the full golden dataset before each release and nightly if you compare multiple models. Also rerun when upstream providers deprecate a model version—silent API changes have broken production apps I maintain.

Ship LLM Features With Confidence

Demos lie; evals tell the truth. When you Evaluate LLM Outputs (Evals) with golden datasets, layered scoring, CI gates, and human spot checks, you catch regressions before users do. That discipline is what separates a chatbot gimmick from a feature worth maintaining. If you want help wiring evals into a Laravel app, RAG pipeline, or client portal, see the custom software development service or review relevant work in the Mijar Law Associates portfolio. For JSON fixture debugging during setup, use the regex tester alongside schema validators. When you are ready to plan an eval suite for your stack, contact us and we will map cases to your highest-risk workflows first.