Rook: Ceph on Kubernetes

By Kokil Thapa | Last reviewed: September 2026

Rook: Ceph on Kubernetes is how teams run production-grade distributed storage inside the cluster instead of renting separate SAN arrays or cloud disks. You get block volumes for databases, a shared filesystem for legacy apps, and S3-compatible object storage from one control plane. That matters on bare-metal clusters, private clouds, and budget-conscious setups where managed EBS-style volumes are unavailable or expensive. This guide walks through architecture, install steps, StorageClasses, and the production checks I use before trusting Ceph with stateful workloads. If you are new to the storage layer itself, start with our Ceph storage fundamentals overview and the Kubernetes persistent volumes guide.

What is Rook: Ceph on Kubernetes and how does it work?

Rook is a Kubernetes operator. It watches custom resources and reconciles a full Ceph cluster on your worker nodes. Ceph handles data placement, replication, and recovery. Kubernetes handles scheduling, networking, and the volume API your apps already use.

The split is clean. Rook owns lifecycle: bootstrap, upgrades, expansion, and health checks. Ceph owns bytes on disk: erasure coding, CRUSH maps, and self-healing when a drive or node fails. Your application still requests a PersistentVolumeClaim. The Ceph CSI plugin provisions a RBD image or CephFS export behind the scenes.

Three Ceph daemons matter day to day. Monitors (MON) hold the cluster map and require an odd count—usually three—for quorum. OSDs store objects on raw devices you assign through Rook. Managers (MGR) expose metrics and the dashboard. RGW adds S3-compatible object storage when you enable it in the CephCluster spec.

On production Linux cluster administration engagements, I treat Rook like any other stateful platform service. Backups, monitoring, and upgrade windows are planned before the first PVC lands on Ceph block storage.

How do you install Rook and deploy a Ceph cluster on Kubernetes?

Start with a Kubernetes cluster that meets Rook’s baseline: a supported version, working CoreDNS, and nodes that can reach each other on pod networks. Rook publishes compatibility matrices on its documentation site—verify your control plane before you install.

Prepare nodes and disks

Ceph wants dedicated disks, not the OS root volume. Attach raw block devices or empty partitions per node. Label nodes if you want storage-only workers:

kubectl label node worker-storage-1 rook-ceph-storage=enabled
kubectl label node worker-storage-2 rook-ceph-storage=enabled
kubectl label node worker-storage-3 rook-ceph-storage=enabled

Verify devices are visible inside the node. Wipe prior filesystem signatures if these disks were reused:

lsblk -f
sudo wipefs -a /dev/sdb

Install the Rook operator

Apply the upstream manifests from the Rook release you pin—do not track master in production:

git clone --depth 1 --branch v1.15.0 https://github.com/rook/rook.git
cd rook/deploy/examples
kubectl create -f crds.yaml -f common.yaml -f operator.yaml

Wait until the operator pod is running in rook-ceph:

kubectl -n rook-ceph get pods -l app=rook-ceph-operator

Create the CephCluster custom resource

Edit cluster.yaml to match your environment. A minimal production-oriented fragment:

apiVersion: ceph.rook.io/v1
kind: CephCluster
metadata:
  name: rook-ceph
  namespace: rook-ceph
spec:
  cephVersion:
    image: quay.io/ceph/ceph:v19.2.0
  dataDirHostPath: /var/lib/rook
  mon:
    count: 3
    allowMultiplePerNode: false
  storage:
    useAllNodes: false
    useAllDevices: false
    nodes:
      - name: worker-storage-1
        devices:
          - name: sdb
      - name: worker-storage-2
        devices:
          - name: sdb
      - name: worker-storage-3
        devices:
          - name: sdb
  dashboard:
    enabled: true
  network:
    provider: host

Apply the cluster and toolbox manifests:

kubectl create -f cluster.yaml
kubectl create -f toolbox.yaml

Watch bootstrap progress. First install can take ten to twenty minutes while monitors form and OSDs come up:

kubectl -n rook-ceph get cephcluster
kubectl -n rook-ceph get pods

Clusters built with Kubespray on bare metal are a common home for Rook. You already own the nodes and disks. Rook fills the storage gap that cloud CSI drivers cannot.

When should you choose Rook Ceph instead of other Kubernetes storage?

Not every cluster needs Ceph. The operator adds operational surface area. You are running distributed storage software on the same nodes that run apps. That trade-off is worth it when requirements exceed what NFS or local volumes provide.

For a multi-service platform like a directory platform with uploads and search, object storage through Ceph RGW can replace a separate MinIO deployment. That reduces moving parts when your team already runs Rook for databases.

How do you expose Ceph storage to applications with StorageClasses and CSI?

After the cluster reaches HEALTH_OK, create StorageClasses for block and file workloads. Rook ships example manifests for RBD block volumes and CephFS shared volumes.

Block storage with RBD

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: rook-ceph-block
provisioner: rook-ceph.rbd.csi.ceph.com
parameters:
  clusterID: rook-ceph
  pool: replicapool
  imageFormat: "2"
  imageFeatures: layering
  csi.storage.k8s.io/provisioner-secret-name: rook-csi-rbd-provisioner
  csi.storage.k8s.io/provisioner-secret-namespace: rook-ceph
  csi.storage.k8s.io/controller-expand-secret-name: rook-csi-rbd-provisioner
  csi.storage.k8s.io/controller-expand-secret-namespace: rook-ceph
  csi.storage.k8s.io/node-stage-secret-name: rook-csi-rbd-node
  csi.storage.k8s.io/node-stage-secret-namespace: rook-ceph
allowVolumeExpansion: true
reclaimPolicy: Delete

Test with a simple PVC and pod before you migrate production databases:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: ceph-test-pvc
spec:
  accessModes: [ReadWriteOnce]
  storageClassName: rook-ceph-block
  resources:
    requests:
      storage: 5Gi

Understand the persistent volume lifecycle before setting reclaimPolicy. Delete removes the RBD image when the PVC disappears. Production databases often need retain policies plus manual cleanup procedures.

Filesystem and object storage

CephFS suits multiple pods that must read/write the same path. Enable a CephFilesystem CR, then provision a CephFS StorageClass. For S3-compatible buckets, use ObjectBucketClaim resources against an RGW endpoint—useful for attachment stores and backup targets.

Snapshot workflows integrate with the Kubernetes volume snapshots API when you install the snapshot controller and define a VolumeSnapshotClass pointing at the Ceph pool. Test restore into a new namespace before you rely on snapshots for disaster recovery.

What production practices keep Rook Ceph healthy on Kubernetes?

Ceph failures are rarely sudden. They announce themselves as slow requests, degraded pools, or near-full OSDs. Operational discipline matters more than exotic tuning.

Capacity and placement

Run at least three storage nodes for replica-size-three pools. Never fill OSDs past seventy-five percent—backfill and recovery need headroom. Set Ceph’s full ratios in the cluster spec or via the toolbox:

kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd set-full-ratio 0.85
kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd set-backfillfull-ratio 0.80

Spread monitors across failure domains. On bare metal, that means different racks or power circuits when possible. Combine with topology spread constraints for app pods—not only for Ceph daemons.

Monitoring, backups, and upgrades

Scrape Ceph metrics through the MGR Prometheus module or the Rook dashboard. Alert on HEALTH_WARN, OSD down counts, and pool near-full conditions before apps time out. Pair metrics with Kubernetes troubleshooting workflows so on-call engineers know whether the app or the volume layer failed first.

Application backups through Velero protect Kubernetes objects and PVC snapshots. They do not replace Ceph-level disaster planning. Export critical pool configuration and practice full cluster rebuild from scratch at least once.

Upgrade Rook during a maintenance window. Bump the operator manifest first, then the CephCluster cephVersion image. Read the release notes on the official Rook documentation before skipping minor versions. Ceph upgrades can trigger data rebalancing—watch latency on production databases during the process.

Lock down the Rook namespace with Kubernetes RBAC. Only platform admins should create or edit CephCluster resources. A mistaken edit can wipe OSD directories or resize pools incorrectly.

Manage manifests through Argo CD GitOps so every change is reviewed. Store cluster specs in Git, pin image tags, and avoid hand-editing live resources. When debugging CRD payloads, paste YAML into the JSON and YAML formatter to catch indentation errors before apply.

Common failure modes

1. OSD pods pending: Usually missing raw devices, wrong deviceFilter, or insufficient memory. Confirm lsblk output matches the CephCluster node list.
2. MON quorum loss: Often network partitions or scheduling two monitors on one failed node. Enforce anti-affinity and keep monitor count odd.
3. Slow PVC provisioning: Check CSI driver logs, pool fullness, and whether secrets in rook-ceph match the StorageClass references.
4. Clock skew: Ceph is sensitive to time drift. Run NTP on every storage node.
5. Bluestore on SSD vs HDD: Mixing tiers without separate pools causes latency spikes. Separate pools and StorageClasses per media type.

Teams building enterprise applications often ask whether Kubernetes belongs in their stack at all. When the answer is yes and workloads are stateful, Rook Ceph becomes the storage foundation that keeps PostgreSQL, Redis persistence, and file imports on-platform rather than on ad-hoc NFS mounts.

Performance tuning overlaps with general Kubernetes performance work: right-size CPU requests for OSD pods, isolate noisy neighbors with taints on storage nodes, and keep etcd on fast disks separate from Ceph data devices. The Ceph documentation remains the authoritative reference for pool parameters, crush rules, and erasure-coded pools when cost efficiency beats replica-three defaults.

For hosting decisions—whether your cluster lives on owned hardware or a provider—see domain and hosting planning alongside long-term support and maintenance contracts. Ceph on three nodes still needs spare drives, monitoring, and someone who responds when a pool enters HEALTH_WARN at midnight.

The Kubernetes storage model itself is documented in the PersistentVolumes concept guide. Rook simply automates what used to require hand-built Ceph Ansible playbooks outside the cluster.

Key Takeaways

• Deploy Rook: Ceph on Kubernetes when you need replicated block, file, and object storage on bare metal or private cloud—not when a cloud CSI already satisfies your SLA.
• Pin Rook and Ceph image versions, dedicate raw disks per OSD node, and validate HEALTH_OK before any production PVC.
• Create separate StorageClasses for RBD block and CephFS shared workloads; test snapshots and restore paths early.
• Keep OSD utilization below seventy-five percent, run three monitors across failure domains, and alert on Ceph health—not only pod restarts.
• Combine Velero application backups with documented Ceph disaster-recovery drills and GitOps-managed manifests.
• Restrict CephCluster edits to platform admins via RBAC; mistaken CR changes can destroy data paths.

People Also Ask

Is Rook Ceph production-ready for Kubernetes?

Yes, for teams that can operate distributed storage. Thousands of clusters run Rook-managed Ceph in production. Success depends on adequate nodes, dedicated disks, monitoring, and upgrade discipline—not on treating Rook as a fire-and-forget add-on.

How many nodes do you need for Rook Ceph?

Plan three or more storage nodes for replica-size-three pools so you survive a single node loss. Lab installs can run reduced setups, but production databases should not rely on two-node compromises that sacrifice quorum or redundancy.

Can Rook Ceph replace cloud block storage?

On infrastructure you control, it often does. On managed Kubernetes where EBS-style volumes are cheap and integrated, cloud CSI is usually simpler. Rook wins when you need unified block, file, and object storage without per-gigabyte cloud fees.

What is the difference between Rook and Ceph?

Ceph is the distributed storage system. Rook is the Kubernetes operator that deploys and manages Ceph daemons using custom resources, CSI drivers, and automated reconciliation—so you do not maintain Ceph entirely outside the cluster lifecycle.

Deploy Rook Ceph with confidence

Rook: Ceph on Kubernetes turns your worker nodes into a unified storage platform for databases, shared files, and object buckets. Start small: one non-production cluster, pinned manifests, a tested PVC, and a snapshot restore drill. Scale out only after monitoring and backup paths are boringly reliable. If you want help designing a stateful platform—from cluster layout to GitOps and ongoing ops—contact us to talk through your storage requirements.