Hubble: Network Observability with eBPF

By Kokil Thapa | Last reviewed: September 2026

Hubble: Network Observability with eBPF answers a question every platform team hits after the first production incident: who talked to whom, on which port, and was it allowed? Traditional packet capture and sidecar proxies add cost and blind spots. Hubble sits on Cilium eBPF networking for Kubernetes and exports flow-level telemetry from the kernel—DNS, TCP, HTTP—without changing your application code. This guide covers architecture, install steps, daily queries, and the mistakes I see on real clusters.

What is Hubble and how does eBPF power network observability?

Hubble is the observability component of the Cilium project. Cilium installs eBPF programs on each node. Those programs hook socket operations, conntrack, and policy enforcement points. Hubble reads the same events Cilium uses for networking and security.

You get flow records instead of raw PCAP files. Each record carries source and destination identities, ports, verdict, and L7 metadata when available. That is fundamentally different from scraping application logs after the fact.

eBPF runs verified bytecode in the kernel. It sees traffic at the point packets and sockets meet the network stack. Sidecars see only what passes through the proxy port. Hubble sees pod-to-pod traffic, host networking, and DNS lookups the moment they happen.

How Hubble differs from classic monitoring

Metrics tell you error rates rose. Traces follow one request path. Hubble shows every connection attempt between identities—allowed, denied, or dropped. That closes the gap when a Kubernetes NetworkPolicy blocks traffic silently.

Hubble complements—not replaces—OpenTelemetry and your APM stack. Use Hubble for network truth. Use OTel for application spans and business context.

How do you install Hubble on a Kubernetes cluster?

Hubble requires Cilium as the CNI. You cannot bolt Hubble onto Calico or Flannel without migrating. Plan a staging cluster first if production already runs another CNI.

On a greenfield cluster, install Cilium with Hubble enabled via Helm. These commands match current Cilium Helm charts as of 2026.

1. Add the Cilium Helm repository and update indexes.
2. Install Cilium with Hubble relay and UI enabled.
3. Verify Cilium pods and Hubble components reach Ready.
4. Port-forward or expose the Hubble UI for first inspection.
5. Install the Hubble CLI locally for scripted queries.

helm repo add cilium https://helm.cilium.io/
helm repo update

helm install cilium cilium/cilium \
  --namespace kube-system \
  --set hubble.enabled=true \
  --set hubble.relay.enabled=true \
  --set hubble.ui.enabled=true \
  --set hubble.metrics.enabled="{dns,drop,tcp,flow,port-distribution,icmp,http}"

kubectl -n kube-system rollout status daemonset/cilium
kubectl -n kube-system get pods -l k8s-app=hubble-relay
kubectl -n kube-system get pods -l k8s-app=hubble-ui

Install the Hubble CLI on your workstation. Binary names vary by OS; pick the release matching your Cilium version.

export HUBBLE_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/cilium/main/stable.txt)
curl -L --remote-name-all \
  "https://github.com/cilium/cilium/releases/download/${HUBBLE_VERSION}/hubble-linux-amd64.tar.gz"
tar xzvf hubble-linux-amd64.tar.gz
sudo mv hubble /usr/local/bin/

Port-forward the UI for a quick smoke test:

kubectl port-forward -n kube-system svc/hubble-ui 8080:80
hubble status

If `hubble status` reports unreachable relay, check NetworkPolicy on `kube-system`. Some hardened clusters block relay gRPC by default. That failure mode is common on first deploy.

Upgrading an existing Cilium cluster

Enable Hubble on a running cluster with a Helm upgrade. Expect a rolling restart of Cilium agents. Schedule during a maintenance window on large node pools.

helm upgrade cilium cilium/cilium \
  --namespace kube-system \
  --reuse-values \
  --set hubble.enabled=true \
  --set hubble.relay.enabled=true \
  --set hubble.ui.enabled=true

After upgrade, confirm flow export before you change production policies. Empty flow lists usually mean relay connectivity or RBAC issues—not missing traffic.

What network data does Hubble expose at each layer?

Hubble classifies flows by observation point and protocol. Understanding those fields saves hours during incident response.

• L3/L4: Source and destination IP, port, protocol, byte and packet counts, TCP flags, verdict (forwarded, dropped, denied).
• Identity: Kubernetes labels resolved to numeric security identities—more stable than IP after pod restarts.
• DNS: Query name, response codes, IPs returned—critical for misconfigured service discovery.
• HTTP: Method, URL path, status code when L7 parsing is enabled and traffic is plain HTTP.
• Drop reasons: Policy denial, CT table full, unsupported protocol—each maps to a distinct troubleshooting path.

Querying flows with the Hubble CLI

The CLI is the fastest path during SSH sessions. Filters accept label selectors, FQDN, verdict, and protocol.

hubble observe --namespace payments \
  --label app=checkout \
  --protocol tcp \
  --verdict DROPPED

hubble observe --namespace kube-system \
  --protocol dns \
  --to-fqdn "*.svc.cluster.local"

Follow mode streams live flows—similar to `tail -f` for connections. Use it while reproducing a bug in staging.

hubble observe --follow --namespace api \
  --label app=gateway

Export JSON for post-incident review or paste into a JSON formatter when sharing with the team.

How does Hubble compare to sidecars, packet capture, and service meshes?

Teams often ask whether Hubble replaces Istio, tcpdump, or cloud flow logs. Each tool answers a different question.

On platforms I maintain, Hubble pairs well with application-level observability for microservices. You do not need a full mesh just to see who blocked port 5432.

For runtime threat detection—not only flow logs—look at Tetragon runtime security with eBPF. Tetragon and Hubble share the Cilium eBPF foundation but serve different operational goals.

How do you export Hubble metrics and integrate with Prometheus?

Flow logs help during incidents. Metrics help you spot drift before users complain. Hubble exposes Prometheus endpoints from each Cilium agent when enabled in Helm values.

Scrape `cilium-agent` pods on port 9965 (default Hubble metrics port). Add a ServiceMonitor if you run the Prometheus Operator.

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: cilium-hubble-metrics
  namespace: monitoring
spec:
  selector:
    matchLabels:
      k8s-app: cilium
  namespaceSelector:
    matchNames:
      - kube-system
  endpoints:
    - port: hubble-metrics
      interval: 30s

Useful metric families include `hubble_flows_processed_total`, DNS query counters, drop counters by reason, and HTTP status histograms. Dashboards in the Cilium repository give starting Grafana panels.

Wire alerts on sustained `drop` verdict growth in critical namespaces. Pair with observability vs monitoring practices so on-call knows which signal owns which runbook.

Long-term flow storage

Hubble Relay holds recent flows in memory—it is not a log warehouse. For retention beyond minutes, pipe flows to Loki, Elasticsearch, or an object store via Hubble export filters.

hubble observe --output json \
  --follow \
  | your-shipper --index kubernetes-flows

Size storage for peak pod churn. Restarts create identity churn even when IPs look stable through services.

What are common Hubble troubleshooting workflows in production?

Most tickets fall into four buckets. Hubble shortens each one if you know which filter to reach for.

1. NetworkPolicy silently blocks traffic

Symptom: application timeout, no error in app logs. Run:

hubble observe --namespace production \
  --pod checkout-abc123 \
  --verdict DROPPED \
  --follow

Dropped flows show the policy verdict and destination identity. Compare against your policy YAML. Fix the label selector—not the app config.

2. DNS failures between namespaces

Filter DNS protocol flows to see NXDOMAIN or timeout patterns.

hubble observe --protocol dns \
  --namespace staging \
  --to-fqdn "database.internal.svc.cluster.local"

This beats guessing Corefile edits. I've used the same pattern on Laravel booking platforms where microservices call internal APIs by short names.

3. Unexpected egress or missing NAT

Identify pods talking to external IPs you did not expect. Export flows and cross-check firewall allow lists on the cloud side.

4. Performance regression after Cilium upgrade

Compare TCP retransmission and drop counters before and after upgrade. Roll back with Helm if drops spike on healthy workloads.

For host-level context outside Kubernetes, combine Hubble findings with Ubuntu network troubleshooting on the node. A healthy CNI path can still fail on iptables-nft mismatches at the host boundary.

Security and compliance notes

Flow logs may contain URL paths and query strings. Treat Hubble exports as sensitive data. Restrict RBAC on Hubble Relay and UI. Mask or drop L7 fields in regulated environments.

Hubble observes traffic; it does not enforce policy alone. Enforcement stays in Cilium network policies and Kubernetes security controls. Read the eBPF overview at ebpf.io if you need the kernel safety model explained for auditors.

Key Takeaways

• Hubble: Network Observability with eBPF requires Cilium—enable relay, UI, and metrics at install or upgrade time.
• Use `hubble observe` with verdict and protocol filters to debug NetworkPolicy and DNS issues in minutes.
• Hubble complements OpenTelemetry and APM; it owns east-west connection truth, not business spans.
• Export Prometheus metrics for drift detection; pipe JSON flows to long-term storage for audits.
• Prefer Hubble over sidecars when you need visibility without per-pod proxy tax—add a mesh only for L7 routing needs.
• Lock down Hubble UI and relay access; flow records can carry sensitive L7 payloads.

People Also Ask

Does Hubble work without Cilium?

No. Hubble reads flow data from Cilium’s eBPF programs. Other CNIs do not expose the same identity-aware flow API. Migrating CNI is the prerequisite.

Can Hubble see HTTPS request bodies?

Hubble sees TLS metadata—SNI, cipher, byte counts—not decrypted HTTP bodies. Plain HTTP L7 parsing works when enabled. For encrypted payload inspection you need mesh termination or application-level tracing.

How is Hubble different from Cilium metrics alone?

Cilium agent metrics aggregate counters. Hubble adds per-flow records, a service map UI, and rich CLI filters. Enable both; they serve different debugging depths.

Is Hubble suitable for multi-cluster observability?

Each cluster runs its own Hubble Relay. Federate Prometheus or ship flow exports to a central store for cross-cluster views. Hubble does not ship a built-in multi-cluster UI as of 2026.

Ship observable networks, not guesswork

Hubble: Network Observability with eBPF turns kernel-level visibility into daily operations tooling. Start on staging: enable relay and UI, run dropped-flow watches during policy changes, and wire Prometheus alerts before production cutover. If you are building or hardening Kubernetes platforms and want help with Cilium, observability pipelines, or Linux system administration, review the portfolio or contact us to talk through your cluster layout.