Build an AI Gateway for LLM Routing

By Kokil Thapa | Last reviewed: September 2026

Every product team that adds chat, summarisation, or document extraction hits the same wall: provider APIs change, costs spike, and one slow model takes down a feature. To Build an AI Gateway for LLM Routing, you put a thin control plane between your app and OpenAI, Anthropic, Google, or self-hosted models. Your Laravel controllers stop calling vendor URLs directly. They call one internal endpoint that picks the model, enforces limits, logs tokens, and fails over when a region blips. If you already ship AI integration and automation services, this pattern is the difference between a demo and something you can run in production for months.

What is an AI Gateway for LLM Routing?

An AI gateway is an API layer that sits between your application and one or more large language model providers. It looks like a single chat-completions endpoint to your frontend or backend. Behind that endpoint, it maps requests to the right upstream model based on rules you define.

Think of it as the same role Kong or Traefik plays for REST microservices, but tuned for LLM workloads. You get one auth model, one logging format, and one place to rotate API keys. Your product code never imports provider-specific SDKs in twelve different controllers.

In my experience working on production Laravel applications, the first LLM feature ships fast. The second one copies the same OpenAI client call. By the third feature, nobody knows which model runs where or what last month cost. A gateway fixes that drift before it becomes operational debt.

Common gateway responsibilities include authentication, per-tenant quotas, prompt and response logging, PII redaction, semantic caching, and circuit breaking when a provider returns 429 or 5xx errors. The routing piece decides which model handles each request. That decision can depend on task type, token budget, user tier, or current provider health.

If you have read about API gateway patterns for microservices, the mental model transfers cleanly. LLM routing adds token accounting, streaming responses, and model capability matrices on top of standard HTTP proxy behaviour.

How do you build an AI Gateway for LLM Routing step by step?

You can buy a managed router such as LiteLLM or Portkey, or you can build a small Laravel 13 service you fully control. For Nepal-based teams with budget constraints, a self-hosted gateway on an existing Ubuntu box often costs less than per-seat SaaS fees. The build path below assumes PHP 8.3+ and Laravel 13.x on a stack you already run for client portals or eCommerce backends.

Step 1: Define a provider-agnostic request contract

Start with one JSON schema your app sends everywhere. Do not leak OpenAI field names into business logic. A minimal contract covers model hint, messages array, max tokens, temperature, stream flag, and metadata tags for billing.

POST /internal/ai/v1/chat
Authorization: Bearer <app-service-token>
Content-Type: application/json

{
  "task": "support_reply",
  "messages": [
    {"role": "system", "content": "You are a helpful support agent."},
    {"role": "user", "content": "Where is my order?"}
  ],
  "max_tokens": 512,
  "metadata": {"tenant_id": "shop-42", "feature": "inbox_assist"}
}

Validate this payload with a Laravel Form Request. Reject oversize prompts before they hit a paid API. A JSON formatter helps you debug malformed bodies during integration.

Step 2: Implement provider adapters

Each adapter translates your internal contract to a vendor HTTP call and maps the response back. Keep adapters thin. They should not contain business rules.

<?php
namespace App\Ai\Providers;

use Illuminate\Support\Facades\Http;

final class OpenAiAdapter implements LlmProvider
{
    public function chat(array $payload): LlmResponse
    {
        $response = Http::withToken(config('ai.openai_key'))
            ->timeout(60)
            ->post('https://api.openai.com/v1/chat/completions', [
                'model' => $payload['upstream_model'],
                'messages' => $payload['messages'],
                'max_tokens' => $payload['max_tokens'],
            ])
            ->throw();

        return LlmResponse::fromOpenAi($response->json());
    }
}

Register adapters in a service container binding keyed by provider name. Anthropic uses a different message format; your adapter normalises that difference. Official docs for the OpenAI Chat Completions API and Anthropic Messages API are the source of truth for field names.

Step 3: Add a routing engine

The router reads config rules and returns a provider plus upstream model. Store rules in config files first. Move to database rows when non-developers need to change routing without deploys.

<?php
namespace App\Ai;

final class LlmRouter
{
    public function resolve(string $task, array $context = []): RouteDecision
    {
        $rules = config('ai.routes');

        foreach ($rules as $rule) {
            if ($rule['task'] !== $task) {
                continue;
            }
            if ($this->matches($rule, $context)) {
                return new RouteDecision(
                    provider: $rule['provider'],
                    model: $rule['model'],
                    fallbacks: $rule['fallbacks'] ?? [],
                );
            }
        }

        return RouteDecision::default();
    }
}

Example config for a legal-tech portal might route short FAQ answers to a cheap model and long document summarisation to a larger one. On projects like Court Marriage In Nepal, that split keeps monthly API spend predictable while quality stays high on complex pages.

Step 4: Wire the gateway controller

One controller orchestrates validate → route → call → log → respond. Queue heavy logging if you expect high volume.

1. Authenticate the caller with Sanctum or a service token.
2. Apply rate limits per tenant and per feature key.
3. Resolve route via LlmRouter.
4. Execute primary provider; on failure, walk the fallback chain.
5. Persist token usage and latency to MySQL or Redis counters.
6. Return a normalised JSON body with model_used and usage fields.

This mirrors how you would structure any internal REST API development project: one entry point, many backends, consistent error shapes.

Which LLM routing strategies work best in production?

Routing is not only picking the cheapest model. You trade cost, latency, capability, and data residency. Pick a primary strategy per feature, then add fallbacks for outages.

Task-based routing fits most business apps. Your booking form sends task: itinerary_draft. Your admin export sends task: csv_cleanup. Each task maps to a rule row. When you add function calling and tool use, route tool-heavy flows only to models that support structured outputs reliably.

Fallback chains deserve explicit design. A pattern I have seen repeatedly: primary OpenAI model, secondary Anthropic model, tertiary local Ollama for degraded mode. Return a clear degraded: true flag so the UI can soften copy when quality drops. Never silently fail into a tiny model without telemetry.

Semantic caching belongs in the gateway, not in each controller. Hash normalised prompts plus task name. Store responses in Redis 8.10 with a TTL that matches content freshness needs. Support portals can cache stable policy answers for hours. Dynamic order-status replies should not cache at all.

Compare build vs buy before you invest weeks. A custom Laravel gateway gives full control and keeps data on your Ubuntu server. Managed routers ship faster and include dashboards. The build vs buy LLM features guide walks through that decision with real trade-offs for small teams.

How do you secure and rate-limit an AI gateway?

Never expose provider API keys to browsers or mobile apps. Only the gateway holds secrets in .env or a vault. Your SPA calls /internal/ai/v1/chat with a short-lived Sanctum token tied to a user or tenant.

Apply layered rate limits:

• Global requests per minute to protect server CPU and outbound bandwidth.
• Per-tenant daily token budget to prevent one client from draining budget.
• Per-feature burst limits for autocomplete-style endpoints.
• Concurrent stream cap so ten tabs cannot open unlimited SSE connections.

Laravel’s rate limiter backed by Redis works well here. Store token counters separately from HTTP request counters. A single chat turn might cost 2,000 tokens while counting as one HTTP hit.

RateLimiter::for('ai-tenant', function (Request $request) {
    return Limit::perMinute(30)->by($request->user()->tenant_id);
});

Log prompts with care. Redact emails, phone numbers, and national ID patterns before write. For legal-tech workflows on portals like Mijar Law Associates, treat uploaded document text as sensitive by default. Follow baseline ideas from AI governance and responsible AI even if you are not enterprise-scale yet.

Put the gateway behind your existing reverse proxy. Terminate TLS at Nginx or Apache, enforce IP allowlists for internal callers, and block public access to admin routes. Patterns from Kong as an API gateway and Amazon API Gateway security apply even when your gateway is a Laravel app, not Kong itself.

How do you monitor costs, latency, and failures?

If you cannot answer “which feature spent the most last week,” you do not have a gateway yet. You have a proxy. Persist one row per call with tenant, task, provider, model, input tokens, output tokens, latency ms, status, and fallback used.

Aggregate nightly into a simple report. Many teams start with a MySQL summary table and a Blade admin page. That beats a fancy dashboard you never finish. Use your EMI calculator mindset here: small regular measurements beat one big surprise bill.

Track these signals from day one:

1. Error rate by provider — catches key expiry and region outages early.
2. p95 latency by model — surfaces slow routes before users complain.
3. Cost per task type — shows where to downgrade models profitably.
4. Cache hit ratio — validates whether semantic cache keys make sense.
5. Fallback frequency — a rising graph means primary routes are unhealthy.

Run periodic evals on routed outputs. Changing the model behind support_reply without regression tests is how quality silently drops. Pair gateway metrics with guidance from how to evaluate LLM outputs and occasional red teaming on high-risk flows.

Alert on budget thresholds in NPR and USD if you serve mixed clients. Rs 50,000 (~USD 375) in unexpected API spend hurts a small agency project. Set Slack or email alerts at 80% of monthly cap. On infrastructure you manage yourself, pair app metrics with server monitoring from Linux system administration practice so CPU spikes from streaming traffic do not masquerade as model slowness.

Deploy the gateway with the same GitLab CI and Deployer 7 workflow you use for other production apps. Reload PHP-FPM after release so opcache picks up routing config changes. Document rollback steps in your runbook alongside normal support and maintenance procedures.

Key Takeaways

• Build an AI Gateway for LLM Routing as a single internal endpoint so apps never call OpenAI or Anthropic directly.
• Use task-based rules plus explicit fallback chains; test every fallback path before launch.
• Rate-limit by tenant and token budget, not only by HTTP request count.
• Log usage per feature from day one so you can cut cost without guessing.
• Keep provider keys on the server, redact PII in logs, and run evals when you change routes.
• Start with config-file routing in Laravel 13; move rules to the database when ops needs self-service control.

People Also Ask

Do I need a separate server for an AI gateway?

Not always. A dedicated Laravel module on your existing app is enough for many SMB products. Split it out when LLM traffic dominates CPU, you need independent scaling, or multiple apps share the same routing policy. Until then, a well-isolated service class and internal route group keeps ops simple.

Can an AI gateway work with local models like Ollama?

Yes. Treat Ollama as another adapter with a base URL and model tag. Route sensitive drafts or offline dev environments to local models. Keep cloud models in the fallback chain for production quality when self-hosted hardware is underpowered.

How is LLM routing different from load balancing?

Load balancing spreads identical requests across replicas of the same service. LLM routing picks different models or providers based on task, cost, capability, or health. You may load-balance two gateway instances behind Nginx, but routing happens inside each instance before the upstream LLM call.

What is the fastest way to prototype routing rules?

Store a YAML or PHP config array mapping task names to provider and model pairs. Wire one PHPUnit test per task that asserts the router returns the expected decision. Expand to admin UI later. Read the AI glossary for engineers if your team mixes up terms like router, proxy, and orchestrator.

Ship a gateway before your second LLM feature

The teams that Build an AI Gateway for LLM Routing early spend less time firefighting vendor outages and surprise invoices. Start with one chat endpoint, three routing rules, and a fallback chain. Add caching and dashboards once traffic proves which tasks matter. If you want help designing routing policy for a Laravel portal, eCommerce site, or internal tool, review the portfolio and about pages, then contact us to talk through your stack.