Tempo: Distributed Tracing with Grafana

By Kokil Thapa | Last reviewed: September 2026

Tempo: Distributed Tracing with Grafana answers a problem every production team hits sooner or later. A checkout fails, a webhook times out, or a queue job stalls — and your logs show fragments, not the full path. Tempo is Grafana’s trace backend. It stores spans cheaply, pairs with Prometheus metrics and Grafana dashboards, and connects to Loki log lines so you can pivot from a spike to the exact request chain. This guide covers architecture, OpenTelemetry wiring, TraceQL, deployment, and how Tempo compares to Jaeger on real stacks.

What is Grafana Tempo and why use it for distributed tracing?

Tempo is a high-volume trace backend built by Grafana Labs. It accepts spans from OpenTelemetry, Jaeger, Zipkin, and other formats. Unlike older systems that index every span attribute up front, Tempo stores compressed blocks on object storage or local disk. You search by trace ID, time range, or TraceQL filters — not by building a costly inverted index of every tag.

That design matters on budget-conscious projects. A law-firm portal or Laravel booking platform may run on a single Ubuntu server or a small Kubernetes cluster. You still need traces when payment callbacks, SMS gateways, or supplier APIs fail. Tempo keeps storage predictable while Grafana gives one UI for metrics, logs, and traces.

The Grafana observability stack — sometimes called the LGTM stack — maps cleanly:

• Prometheus — counters, histograms, and alerts on latency and error rates.
• Loki — structured and unstructured logs with label-based filtering.
• Tempo — distributed traces showing service-to-service timing.
• Grafana — dashboards, Explore, and correlated drill-down.

On stacks I maintain, that correlation saves hours. You see a latency spike on a histogram panel, click an exemplar, and land on the slow trace — then jump to Loki for the same request ID. That workflow beats grep-ing three log files on a shared EC2 host.

How do you send traces to Tempo from OpenTelemetry and PHP apps?

Most greenfield setups use the OpenTelemetry specification and send OTLP to an OpenTelemetry Collector. The collector batches, filters, and forwards spans to Tempo. Direct app-to-Tempo works for small installs, but the collector gives you sampling, attribute scrubbing, and multi-backend export without redeploying code.

Run Tempo with Docker Compose

For a lab or staging server, start with the official single-binary config. Create tempo.yaml:

server:
  http_listen_port: 3200

distributor:
  receivers:
    otlp:
      protocols:
        grpc:
          endpoint: 0.0.0.0:4317
        http:
          endpoint: 0.0.0.0:4318

storage:
  trace:
    backend: local
    local:
      path: /var/tempo/traces
    wal:
      path: /var/tempo/wal

compactor:
  compaction:
    block_retention: 168h

Pair it with a collector config that forwards OTLP to Tempo:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317

processors:
  batch:
    timeout: 5s
    send_batch_size: 1024

exporters:
  otlp/tempo:
    endpoint: tempo:4317
    tls:
      insecure: true

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp/tempo]

Instrument Laravel and PHP services

PHP auto-instrumentation has improved, but production Laravel apps often need deliberate span boundaries. Wrap controller actions, queue jobs, and outbound HTTP calls. Use the OpenTelemetry PHP SDK or run a sidecar collector on the same host as PHP-FPM.

A practical pattern on API-heavy Laravel projects:

1. Generate a trace context at the HTTP middleware layer and attach it to the request.
2. Create child spans for database queries, cache calls, and payment gateway requests.
3. Propagate the traceparent header on internal service calls and webhooks.
4. Export spans via OTLP to the local collector on port 4317.
5. Store the trace ID in your log context so Loki queries align with Tempo.

Validate propagation before you tune sampling. A broken header means each service starts a new trace — useless for debugging cross-service checkout flows.

Use a JSON formatter to inspect OTLP payloads during development. Broken attribute types or oversized span batches are easier to catch locally than in production.

How do you query traces in Grafana with TraceQL?

Tempo’s query language is TraceQL. It filters traces by span attributes, duration, service name, and status — similar in spirit to PromQL for metrics. You run TraceQL in Grafana Explore or embed results in dashboards.

Essential TraceQL patterns

{ resource.service.name = "checkout-api" && duration > 500ms }

{ name = "POST /api/payment/callback" && status = error }

{ span.http.status_code = 502 && resource.service.name = "gateway" }

TraceQL shines when paired with logs and metrics. Configure derived fields in Loki so a log line’s trace ID becomes a link to Tempo. Enable exemplars on Prometheus histograms so Grafana passes the trace ID into Explore.

For dashboard work, see the companion guide on building Grafana dashboards. Trace panels and service graphs turn TraceQL results into operational views your team can scan during incidents.

Service graph and metrics generator

Tempo can generate service graph metrics and span metrics from incoming traces. Those metrics land in Prometheus and power dependency maps — which service calls which, and how often errors occur. That closes a gap when you have traces but no pre-defined RED metrics for every internal endpoint.

Enable the metrics generator in Tempo config and point it at a Prometheus remote-write endpoint. Start with default span metrics; tune cardinality before you label every custom attribute.

How does Tempo compare to Jaeger and Zipkin for production?

Teams evaluating backends usually weigh Tempo against Jaeger and Zipkin. All three accept OpenTelemetry. The differences are storage cost, query model, and Grafana integration depth.

If you already run Prometheus and Loki, Tempo is the natural third pillar. Jaeger remains excellent when you need deep tag search across billions of spans and already operate its storage tier. For a deeper OpenTelemetry walkthrough, read the guide on distributed tracing with OpenTelemetry and Jaeger — the instrumentation steps overlap; only the exporter endpoint changes.

How do you deploy and operate Tempo in production?

Tempo scales from a single binary on Ubuntu to a microservices deployment on Kubernetes. Match the mode to your traffic and ops capacity — not to a diagram you saw at a conference.

Single-server deployment

On a VPS running Laravel, MySQL, Redis, and Grafana, run Tempo as a systemd service or Docker container. Mount a dedicated volume for WAL and trace blocks. Retain seven to fourteen days of traces unless compliance requires longer — disk fills faster than teams expect.

I've encountered this during production deployments on shared EC2 hosts. Traces without retention policy consumed the same partition as MySQL binlogs. Set block_retention and monitor disk with your existing Prometheus node exporter.

Kubernetes and object storage

At higher volume, use S3-compatible storage (AWS S3, MinIO, or similar) as Tempo’s backend. Run distributors, ingesters, queriers, and compactors as separate components. The Grafana Tempo documentation publishes Helm charts and sizing guidance.

Wire Prometheus and Grafana monitoring to Tempo itself — ingest lag, compactor errors, and query latency deserve alerts like any other service.

Sampling and cost control

Tracing every request at 10k RPS is expensive. Use head sampling in the collector for baseline coverage. Add tail sampling for errors and slow requests — keep all traces above 500 ms or with non-2xx status codes.

Scrub PII in the collector before spans reach Tempo. Payment tokens, national ID numbers, and raw email addresses do not belong in span attributes. I've seen teams ship traces faster than they ship log redaction — fix that in the collector pipeline.

What production pitfalls should you avoid with Tempo?

Tempo is operationally simpler than a full Jaeger plus Elasticsearch cluster. It still has sharp edges.

• High-cardinality attributes — user IDs on every span explode metrics generator output. Prefer coarse labels.
• Missing trace context on async jobs — Laravel queue workers need explicit context injection or traces break at the job boundary.
• No link between logs and traces — without trace ID in log lines, Loki and Tempo stay siloed.
• 100% sampling in production — start at 5–10% and raise only when storage budget allows.
• Forgetting compactor monitoring — stuck compactors mean growing WAL and eventual ingest failure.

For ongoing tuning, testing and optimization and support and maintenance contracts often include observability hardening — not because Tempo is exotic, but because teams defer it until the first bad incident.

On client portals with document uploads and payment flows, I treat tracing like backups. You do not skip it because the site is small. You size retention and sampling for the budget you have.

Key Takeaways

• Tempo stores traces with minimal indexing — pair it with Grafana, Prometheus, and Loki for full observability.
• Send spans via OpenTelemetry Collector over OTLP; validate trace context propagation before tuning sampling.
• Query with TraceQL in Grafana Explore; link logs and metrics through trace IDs and exemplars.
• Choose Tempo over Jaeger when Grafana is already your ops hub and storage cost matters at scale.
• Set retention, sampling, and PII scrubbing on day one — disk and compliance problems arrive quietly.
• Alert on Tempo compactor health and ingest lag the same way you monitor application services.

People Also Ask

Does Tempo replace Jaeger entirely?

Not always. Tempo replaces Jaeger as your trace storage backend when you want Grafana-native correlation and lower indexing overhead. Jaeger’s UI still suits teams that rely on deep tag search and already run Jaeger storage. Many migrations keep Jaeger instrumentation and point OTLP export at Tempo instead.

Can Tempo run without Kubernetes?

Yes. The single-binary mode runs on a plain Linux server with local disk or S3-compatible storage. Several stacks I maintain use Docker Compose on Ubuntu alongside PHP-FPM and MySQL — no cluster required for moderate trace volume.

How long should you retain traces in Tempo?

Seven to fourteen days covers most incident response needs. Extend retention when compliance or post-mortem policy requires it — and move blocks to object storage rather than filling local SSD. Adjust block_retention and compactor settings to match.

What is the difference between Tempo and Loki?

Loki stores log lines indexed by labels. Tempo stores distributed trace spans grouped by trace ID. They complement each other in Grafana — logs tell you what was said, traces show you the path and timing across services. Link them with shared trace IDs in log fields.

Ship tracing before the next outage

Tempo: Distributed Tracing with Grafana gives you request-level visibility without running a separate observability silo. Instrument with OpenTelemetry, forward through a collector, store in Tempo, and query in Grafana alongside the metrics and logs you already have. Start on staging, verify propagation through your slowest workflow — payments, webhooks, or queue chains — then roll sampling and retention into production.

Need help wiring Tempo into a Laravel stack or your existing Prometheus setup? Contact us for observability and Linux system administration support. Browse the portfolio for production platforms that depend on reliable monitoring, or read more on API monitoring with Prometheus and Grafana and Alertmanager alerting patterns.