KServe: Model Serving on Kubernetes

By Kokil Thapa | Last reviewed: September 2026

You are building an AI/ML platform on Kubernetes with gRPC inference endpoints. Traffic is not a burst of short REST calls — it is long-lived streaming connections, per-model routes, and latency-sensitive GPU work. A plain Ingress controller and ClusterIP Service will not give you per-route metrics, safe autoscaling signals, or stable bidirectional streams under pod churn. You need a service mesh layer, and usually a model-serving framework like KServe: Model Serving on Kubernetes on top of it. The sections below compare the top mesh options against those three requirements so you can pick infrastructure that survives production load, not just a demo cluster.

On production systems I maintain, the serving layer sits beside traditional web APIs. The same discipline applies: define contracts, measure per route, and test failure modes before customers hit them. If you already ship HTTP APIs, start from Laravel API best practices and extend the pattern to gRPC. KServe standardizes model deployment through an InferenceService CRD. The mesh handles connection lifecycle, mTLS, and observability underneath. Skip either layer and you will debug 503s during scale events instead of shipping models.

What service mesh options handle gRPC streaming, per-route metrics, and autoscaling on Kubernetes?

Not every mesh treats gRPC as a first-class protocol. HTTP/2 multiplexing, long-lived streams, and protobuf payloads need proxy support for streaming RPCs, request classification by method or header, and metrics that do not collapse everything into one generic counter. For AI/ML inference, you also need autoscaling that reads concurrency or custom queue metrics — not just average CPU, which stays flat while request queues grow.

The practical shortlist for 2026 clusters:

• Istio — Richest per-route telemetry, VirtualService traffic splits, and native integration with KServe's default networking path. Higher sidecar cost per pod.
• Linkerd — Minimal Rust proxy overhead; strong for stable unary and server-streaming gRPC where you want simplicity over advanced routing.
• Cilium — eBPF dataplane with optional Envoy L7 policy; excellent throughput on GPU-heavy nodes where sidecar RAM matters.
• Consul Connect — Worth considering if you already run HashiCorp Consul for service discovery outside Kubernetes.

KServe itself is not a mesh. It is the model-serving control plane. In Knative mode it expects Knative Serving, which in turn expects Istio or compatible ingress. In "raw deployment" mode you can run KServe with Istio, Linkerd, or Cilium as the data plane. That split is the architecture decision most teams get wrong on the first pass.

How does KServe: Model Serving on Kubernetes fit with your mesh choice?

KServe solves the last mile of ML deployment: turn a model artifact in S3, GCS, or a PVC into a versioned, autoscaled endpoint. It exposes REST and gRPC through the KServe V2 inference protocol. That protocol maps cleanly to gRPC for low-latency token streaming in LLM workloads.

The architecture has three layers. The InferenceService controller reconciles desired state. The mesh or Knative layer routes traffic and enforces TLS. Runtime pods — Triton, TorchServe, MLServer, or custom containers — execute inference. You can upgrade Triton without touching Istio VirtualServices. That separation is why teams already on Istio service mesh adopt KServe faster than greenfield mesh installs.

A minimal production manifest for a scikit-learn classifier shows the pattern. Note explicit resources and a model-specific readiness probe — not a generic /healthz check.

apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  name: sklearn-doc-classifier
  namespace: ml-production
  annotations:
    prometheus.io/scrape: "true"
    prometheus.io/port: "8080"
    serving.kserve.io/enable-prometheus-scraping: "true"
spec:
  predictor:
    minReplicas: 2
    maxReplicas: 10
    scaleTarget: 70
    sklearn:
      storageUri: "s3://ml-models/doc-classifier/v2.4.1/model.joblib"
      resources:
        requests:
          cpu: "500m"
          memory: "512Mi"
        limits:
          cpu: "2"
          memory: "2Gi"
      readinessProbe:
        httpGet:
          path: /v1/models/sklearn-doc-classifier/ready
          port: 8080
        initialDelaySeconds: 30
        periodSeconds: 10

Models often need 30–90 seconds to load after the container starts. If readiness passes before weights are in memory, the mesh routes gRPC streams to pods that return 503 until load completes. Always probe the runtime's model-ready endpoint. For private registries, reference credentials through Kubernetes secrets via storageConfig — never embed keys in the spec. On self-hosted MinIO clusters, set the endpoint URL with protocol and port explicitly; the default S3 client assumes AWS endpoints.

Enabling gRPC on the InferenceService

For gRPC clients, expose port 8081 (gRPC) alongside 8080 (REST) on supported runtimes. Configure your mesh VirtualService or Gateway to route application/grpc traffic to the gRPC port. With Istio, a typical destination rule sets load balancing to LEAST_REQUEST for inference — round-robin spreads load evenly but ignores queue depth inside each pod.

apiVersion: networking.istio.io/v1
kind: VirtualService
metadata:
  name: sklearn-doc-classifier-grpc
spec:
  hosts:
    - sklearn-doc-classifier.ml-production.svc.cluster.local
  http:
    - match:
        - port: 8081
      route:
        - destination:
            host: sklearn-doc-classifier-predictor
            port:
              number: 8081
          weight: 100

Validate stream behavior with gRPC keepalive settings on both client and server. A common production failure: the mesh idle timeout closes a stream the client still considers open. Set Istio idleTimeout above your longest expected stream, or enable application-level pings every 30 seconds.

Which service mesh is best for long-lived gRPC streaming connections?

Streaming inference — token-by-token LLM output, continuous embedding feeds, or bidirectional audio — keeps HTTP/2 connections open for minutes. Each mesh handles that differently.

Istio remains the default for KServe deployments that use Knative. You get per-revision traffic splits for canary model rollouts, mTLS between transformer and predictor pods, and Grafana dashboards from observability with a service mesh without custom instrumentation in every model container. The cost is sidecar RAM on every pod — painful on small GPU nodes where memory is already tight.

Linkerd wins when you want mesh benefits without Istio's operational surface. I've seen Linkerd sidecars add under 30 Mi per pod versus 150 Mi for Envoy. For server-streaming gRPC where you rarely need complex fault injection, that savings matters at 200+ replica counts. Per-route metrics exist but with fewer label dimensions than Istio.

Cilium suits clusters already standardized on eBPF networking. Cilium eBPF networking for Kubernetes moves L3/L4 policy into the kernel. Enable Envoy for L7 gRPC routing when you need method-level policy. Hubble gives flow-level visibility that complements Prometheus scrape targets on KServe predictors.

How do you get per-route metrics for gRPC inference endpoints?

CPU metrics tell you the node is busy. They do not tell you that /inference.GRPCInferenceService/ModelStreamInfer on model llama-70b-v3 has p99 latency of 8 seconds while embed-v2 stays under 200 ms. Per-route visibility requires protocol-aware proxies or application instrumentation.

Mesh-native metrics

Istio emits istio_requests_total and istio_request_duration_milliseconds with destination_service, response_code, and gRPC status codes. Scrape the sidecar stats endpoint or use the Istio telemetry API. For KServe, also scrape predictor metrics: kserve_inference_request_total, kserve_inference_latency_microseconds, and kserve_model_load_time.

Wire both into OpenTelemetry and Grafana. Tag dashboards by InferenceService name and revision label so canary comparisons are one panel away. Export mesh spans through instrumenting apps with OpenTelemetry when you need end-to-end traces from API gateway to GPU pod.

Application-level metrics for streaming

Mesh proxies measure connection duration well. They miss token throughput or time-to-first-token on LLM streams. Add application counters in your predictor or a KServe transformer sidecar:

• inference_tokens_total{model, revision} — counter for streamed tokens
• inference_time_to_first_token_seconds — histogram for UX-sensitive SLIs
• inference_active_streams — gauge for autoscaling beyond CPU

Use the JSON formatter tool to inspect V2 protocol payloads during development. Misaligned tensor shapes fail silently in streaming mode until clients timeout. Catch schema errors before they reach production metrics dashboards.

How should autoscaling work for gRPC inference on Kubernetes?

Standard Horizontal Pod Autoscaler on CPU fails for GPU inference. A loaded GPU shows 95% utilization at one concurrent stream. Ten queued requests still read as "healthy" until latency explodes. KServe and Kubernetes offer better signals.

Knative Pod Autoscaler (KPA) scales on concurrent requests per pod. Set containerConcurrency: 1 for GPU models — one active stream per replica. Set minReplicas: 2 for latency-sensitive gRPC services where scale-from-zero cold starts (5–15 seconds plus model load) break SLAs. See Kubernetes autoscaling: HPA, VPA, and cluster autoscaler for how cluster-level scaling interacts with pod-level decisions.

KEDA fits batch and queue-driven inference. Scale on Prometheus queries like sum(inference_active_streams{model="embed-v2"}) > 50 or Kafka consumer lag. KEDA event-driven autoscaling decouples scale triggers from the mesh entirely — the mesh routes; KEDA decides replica count from business metrics.

Worked example: An embedding service targets 30 concurrent gRPC streams per pod at p99 under 400 ms. At 120 active streams, KPA adds three replicas. If each stream lasts 90 seconds (long document chunking), short CPU spikes mislead HPA into under-scaling. Track inference_active_streams instead.

For models over 1 GB, enable warmup in the predictor spec or stage weights on a node-local PVC. Warmup cut first-request p99 from 4.2 s to 380 ms on a document embedding service I debugged — same cluster, same mesh, just pre-loaded JIT state before the Service marked the pod ready.

How do KServe, Seldon Core, and BentoML compare for production?

The mesh handles connections and metrics. The serving framework handles model lifecycle. All three below run on Kubernetes; only the operational model differs.

If you already run Istio, KServe is the path of least resistance. Seldon Core fits when preprocessors, predictors, and explainers chain in one graph with drift detection. BentoML fits Python-heavy teams that may not want Knative's complexity. There is no universal winner — only alignment with your mesh and team skills. Treat each model version like a microservice boundary, the same way microservice migration strategies define contracts and rollback for HTTP services.

For GPU scheduling, node selectors, and tensor parallelism, see running AI/ML workloads on Kubernetes with GPUs. For pipeline orchestration upstream of serving, Kubeflow ML pipelines on Kubernetes and MLOps from notebook to production cover the path from training artifact to InferenceService spec.

What production pitfalls break gRPC inference platforms on Kubernetes?

Documentation covers happy paths. Production breaks on edge cases. These five issues recur across clusters.

1. Istio sidecar resource starvation: Default proxy limits are 100–200 Mi RAM per pod. On a 16 Gi GPU node running four inference replicas, sidecars alone consume 800 Mi before model weights load. Set proxyCPU and proxyMemory explicitly in Istio ProxyConfig.
2. Stream timeout mismatch: Client keepalive at 30 s but Envoy stream_idle_timeout at 5 min still fails if a load balancer upstream closes at 60 s. Trace the full chain: client → external LB → mesh gateway → sidecar → pod.
3. Model download timeouts: Init containers pulling 2 GB+ artifacts exceed default deadlines. Pre-stage on PVC or raise initContainerTimeout in the KServe configmap. A production booking platform mindset applies — stage assets before traffic arrives, same as warming cache before a sale event.
4. Autoscaling on CPU for GPU models: HPA sees 40% CPU and holds replicas flat while gRPC queue depth grows. Switch to concurrency or custom Prometheus metrics via KEDA.
5. NetworkPolicy blocking internal traffic: Zero-trust policies that block predictor-to-transformer traffic on ports 8080, 8081, and 9090 cause opaque 503 errors. Whitelist the serving namespace explicitly.

Also validate V2 protocol compliance in custom transformers. KServe defaults to the Open Inference Protocol. Clients sending V1-shaped JSON to a gRPC V2 endpoint fail silently until timeouts. Use gRPC vs REST for service-to-service and REST vs GraphQL vs gRPC guides to align client SDK choices with your mesh routing rules.

Key Takeaways

• For gRPC inference on Kubernetes, Istio + KServe is the default when you need per-route metrics, canary rollouts, and Knative autoscaling together.
• Linkerd and Cilium are strong alternatives when sidecar overhead or eBPF throughput matter more than Istio's routing depth.
• Autoscale on concurrent streams or custom Prometheus gauges — not CPU alone — especially for GPU-backed models.
• Tune idle timeouts and keepalive at every hop so long-lived streaming connections survive load balancer and mesh proxies.
• Scrape both mesh metrics and KServe predictor metrics; add time-to-first-token counters for LLM streams.
• Start with one non-critical model, validate gRPC stream behavior under pod churn, then migrate customer-facing endpoints.

People Also Ask

Do I need a service mesh if I only expose KServe through an Ingress controller?

For a single model and REST-only traffic, a well-tuned Ingress plus ClusterIP Service may suffice. Once you run multiple models, gRPC streaming, mTLS between components, or canary revisions, a mesh pays for itself in observability and safe rollouts. KServe's Knative mode assumes mesh-capable ingress anyway.

Can Linkerd handle KServe canary deployments?

Yes, through SMI TrafficSplit resources targeting KServe revision Services. You get percentage-based splits without Istio VirtualServices. Advanced fault injection and header-based routing remain Istio's strength.

Why does gRPC autoscaling fail with standard HPA?

HPA reads CPU or memory averages. GPU inference keeps CPU low while request queues grow. gRPC streams are long-lived, so "requests per second" undercounts active load. Use Knative concurrency targets or KEDA with a Prometheus query on active streams.

Is KServe still the right choice without Knative?

Yes. KServe supports raw Kubernetes Deployment mode with Istio, Linkerd, or Cilium. You lose scale-to-zero but keep the InferenceService abstraction, multi-runtime support, and V2 gRPC protocol. Many production GPU clusters disable scale-to-zero intentionally.

Build your gRPC inference platform with the right mesh and serving stack

Building an AI/ML platform on Kubernetes with gRPC inference endpoints comes down to two choices: which service mesh handles your streaming connections and per-route metrics, and which serving framework — KServe: Model Serving on Kubernetes is the strongest default on Istio — manages model lifecycle and autoscaling signals. Start small, measure time-to-first-token and stream error rates from day one, and document rollback before you need it.

If you are integrating ML inference alongside existing web platforms — payment APIs, booking flows, document portals — AI integration and automation services can help you design a stack your team can operate after launch. For architecture review or implementation support, contact us or reach out directly to discuss your cluster constraints, mesh choice, and serving requirements.