Prometheus and Grafana: Full Monitoring Stack

By Kokil Thapa | Last reviewed: August 2026

Setting up a reliable Prometheus and Grafana: Full Monitoring Stack is the difference between guessing why your Laravel application slowed down during peak hours and knowing exactly which database query caused the bottleneck. For developers managing production infrastructure in Nepal or remotely, relying solely on basic uptime checks or shared hosting dashboards is insufficient for debugging complex performance issues. This guide walks through a battle-tested deployment on Ubuntu 24.04 LTS, focusing on practical security, resource management, and application-level integration rather than theoretical architecture.

How do you architect the Prometheus and Grafana full monitoring stack?

Understanding the data flow prevents misconfiguration later. In my experience working on production Laravel applications, treating monitoring as an afterthought leads to gaps where critical metrics are missing during incidents. The DevOps automation principles apply directly here: every component must be declarative and reproducible.

Prometheus operates on a pull model, actively scraping HTTP endpoints at defined intervals. This contrasts with push-based systems and has significant implications for firewall rules and service discovery. Grafana connects to Prometheus purely as a read-only data source; it never writes metrics back. On a typical Laravel project, I run Node Exporter for OS-level metrics (CPU, memory, disk I/O) alongside a PHP-FPM exporter and an application-specific endpoint exposing business metrics like queue depth or payment processing latency.

Component responsibilities

• Prometheus: Scrapes targets every 15–60 seconds, evaluates alerting rules, stores compressed time-series blocks on disk.
• Grafana: Queries PromQL against the Prometheus HTTP API, renders visualizations, manages user authentication and dashboard permissions.
• Node Exporter: Exposes hardware and OS metrics from /proc and /sys without requiring root privileges when configured correctly.
• Application Exporters: Custom or community-maintained adapters that translate application state into Prometheus exposition format.

How do you install and configure Prometheus on Ubuntu 24.04?

Avoid installing Prometheus via apt unless you understand the version lag. The official repositories often carry outdated releases missing recent security patches or PromQL functions. Download the latest stable binary (2.53.x as of mid-2026) directly from GitHub releases. This approach also simplifies upgrades and rollback.

# Create system user and directories
sudo useradd --no-create-home --shell /bin/false prometheus
sudo mkdir -p /etc/prometheus /var/lib/prometheus
sudo chown prometheus:prometheus /etc/prometheus /var/lib/prometheus

# Download and extract (verify checksum in production)
wget https://github.com/prometheus/prometheus/releases/download/v2.53.1/prometheus-2.53.1.linux-amd64.tar.gz
tar xvfz prometheus-2.53.1.linux-amd64.tar.gz
sudo cp prometheus-2.53.1.linux-amd64/prometheus /usr/local/bin/
sudo cp prometheus-2.53.1.linux-amd64/promtool /usr/local/bin/
sudo chown prometheus:prometheus /usr/local/bin/prometheus /usr/local/bin/promtool

The configuration file determines what gets monitored. A common mistake in tutorials is omitting scrape timeouts or setting unrealistic intervals. For most web applications, a 30-second interval balances freshness with resource consumption. Always validate configuration before reloading:

# /etc/prometheus/prometheus.yml
global:
  scrape_interval: 30s
  evaluation_interval: 30s
  scrape_timeout: 10s

rule_files:
  - "alerts/*.yml"

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  - job_name: 'node-exporter'
    static_configs:
      - targets: ['localhost:9100']

  - job_name: 'laravel-app'
    metrics_path: '/metrics'
    scheme: https
    authorization:
      type: Bearer
      credentials_file: /etc/prometheus/bearer_token
    static_configs:
      - targets: ['your-domain.com']

Create the systemd unit file to manage the service. Note the explicit storage path and retention flags — these prevent disk exhaustion on smaller VPS instances commonly used in Nepal:

# /etc/systemd/system/prometheus.service
[Unit]
Description=Prometheus Monitoring System
Wants=network-online.target
After=network-online.target

[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/prometheus \
  --config.file=/etc/prometheus/prometheus.yml \
  --storage.tsdb.path=/var/lib/prometheus \
  --storage.tsdb.retention.time=30d \
  --storage.tsdb.retention.size=8GB \
  --web.console.templates=/etc/prometheus/consoles \
  --web.console.libraries=/etc/prometheus/console_libraries \
  --web.enable-lifecycle

Restart=always
RestartSec=5

[Install]
WantedBy=multi-user.target

Validating and starting the service

1. Run sudo -u prometheus promtool check config /etc/prometheus/prometheus.yml to catch syntax errors.
2. Execute sudo systemctl daemon-reload && sudo systemctl enable --now prometheus.
3. Verify status with systemctl status prometheus and check journalctl -u prometheus -f for startup warnings.
4. Access http://localhost:9090/targets to confirm all scrape targets show UP status.

How do you secure Prometheus and Grafana endpoints in production?

This is where most tutorials fail catastrophically. Exposing unauthenticated Prometheus or Grafana to the public internet leaks sensitive infrastructure data and creates attack vectors. In my experience maintaining legal-tech portals handling sensitive documents, security cannot be optional. Reference the server security hardening guide for foundational UFW and fail2ban setup before proceeding.

Bind services to localhost only

Never let Prometheus or Grafana listen on 0.0.0.0. Modify the systemd unit for Prometheus to include --web.listen-address=127.0.0.1:9090. For Grafana, edit /etc/grafana/grafana.ini:

[server]
http_addr = 127.0.0.1
http_port = 3000
domain = your-domain.com
root_url = %(protocol)s://%(domain)s:%(http_port)s/grafana/
serve_from_sub_path = true

Nginx reverse proxy with authentication

Use HTTP Basic Auth as a minimum barrier. Generate credentials with htpasswd -c /etc/nginx/.htpasswd monitor_user. The following Nginx configuration assumes Let's Encrypt certificates are already provisioned:

server {
    listen 443 ssl http2;
    server_name monitor.your-domain.com;

    ssl_certificate /etc/letsencrypt/live/monitor.your-domain.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/monitor.your-domain.com/privkey.pem;

    location /prometheus/ {
        auth_basic "Monitoring Restricted";
        auth_basic_user_file /etc/nginx/.htpasswd;
        proxy_pass http://127.0.0.1:9090/;
        proxy_set_header Host $host;
    }

    location /grafana/ {
        auth_basic "Monitoring Restricted";
        auth_basic_user_file /etc/nginx/.htpasswd;
        proxy_pass http://127.0.0.1:3000/;
        proxy_set_header Host $host;
    }
}

For higher-security environments, replace Basic Auth with OAuth2-proxy or Cloudflare Access. IP whitelisting via UFW provides defense-in-depth but should not be the sole protection mechanism since IP addresses can be spoofed or change unexpectedly.

How do you integrate Laravel application metrics with Prometheus?

Infrastructure metrics alone don't tell you if your business logic is healthy. For Laravel applications, expose custom metrics using the spatie/laravel-prometheus package or the lower-level promphp/prometheus_client_php library. This bridges the gap between server health and application behavior, which is critical when debugging issues reported by clients using platforms like legal case tracking systems where downtime directly impacts court deadlines.

// Install via Composer
composer require spatie/laravel-prometheus

// Publish config
php artisan vendor:publish --tag=prometheus-config

// In app/Providers/AppServiceProvider.php boot() method
use Spatie\Prometheus\Collectors\Laravel\QueueCollector;
use Spatie\Prometheus\Collectors\Laravel\CacheCollector;

public function boot(): void
{
    Prometheus::registerCollector(new QueueCollector());
    Prometheus::registerCollector(new CacheCollector());
    
    // Custom business metric example
    Prometheus::registerCollector(
        new class implements Collector {
            public function collect(): array
            {
                return [
                    Metric::gauge('pending_attestation_requests')
                        ->help('Number of attestation requests awaiting review')
                        ->value(fn() => AttestationRequest::whereStatus('pending')->count()),
                ];
            }
        }
    );
}

Essential Laravel metrics to track

Protect the /metrics endpoint with bearer token authentication or IP restriction. Never expose it publicly. Add middleware to validate the token matches the credential file referenced in your Prometheus scrape configuration.

How do you configure alerting rules and Grafana dashboards effectively?

Dashboards without alerts create passive monitoring that fails during incidents. Define alerting rules in Prometheus itself, not just in Grafana, to ensure alerts fire even if Grafana is down. Store rules in separate YAML files under /etc/prometheus/alerts/:

# /etc/prometheus/alerts/laravel.yml
groups:
  - name: laravel-application
    rules:
      - alert: HighQueueBacklog
        expr: laravel_queue_jobs_pending > 100
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Queue backlog exceeding threshold"
          description: "{{ $labels.job }} has {{ $value }} pending jobs for 5 minutes"

      - alert: LowCacheHitRate
        expr: rate(laravel_cache_hits_total[5m]) / rate(laravel_cache_operations_total[5m])  0.7
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "Cache hit rate below 70%"
          description: "Application may be experiencing excessive database load"

Building useful Grafana dashboards

Import community dashboards as starting points, then customize ruthlessly. Dashboard ID 1860 (Node Exporter Full) and 11277 (Laravel Stats) cover 80% of needs. However, always add a top-level row with business KPIs relevant to your specific application. For eCommerce sites, this means order conversion rates and payment gateway error counts. For legal portals, document submission volumes and average processing times matter more than generic CPU graphs.

Configure Grafana provisioning to make dashboards and data sources version-controlled. Place YAML files in /etc/grafana/provisioning/dashboards/ and /etc/grafana/provisioning/datasources/. This ensures your monitoring stack survives server rebuilds and can be replicated across staging and production environments using the same CI/CD pipeline patterns used for application code.

What retention and performance tuning prevents disk exhaustion?

Prometheus stores raw samples for the configured retention period, then compacts them into larger blocks. Without explicit limits, a busy server can consume all available disk space within weeks. Set both time-based and size-based retention in the systemd unit as shown earlier. The size limit acts as a safety valve when unexpected metric cardinality explosions occur.

Monitor Prometheus's own internal metrics to detect problems early. Key indicators include prometheus_tsdb_head_chunks (memory pressure), prometheus_tsdb_compactions_failed_total (disk I/O issues), and prometheus_target_scrape_pool_sync_total (configuration reload problems). If head chunks consistently exceed 1 million on a 4GB RAM server, reduce scrape frequency or drop high-cardinality labels.

For long-term retention beyond 30 days, consider Thanos or Mimir as object-storage-backed solutions. However, for most small-to-medium deployments in Nepal, rotating 30-day local retention with weekly offsite backups of the /var/lib/prometheus directory provides adequate history without operational complexity. Test restore procedures quarterly — backups you cannot restore are worthless.

Deploying Your Monitoring Stack With Confidence

The Prometheus and Grafana: Full Monitoring Stack gives you ownership over observability without vendor lock-in or recurring SaaS fees. Start with the core three components (Prometheus, Grafana, Node Exporter), secure them properly behind Nginx, and add application metrics incrementally based on actual incident post-mortems. Resist the urge to instrument everything upfront; monitor what hurts first. If you need assistance implementing this stack for your Laravel application or configuring production-grade alerting for Nepal-based infrastructure, reach out to discuss your monitoring requirements.