Deploying CephFS as a Self-Hosted GCP Filestore Alternative

CephFS is a POSIX-compliant distributed file system built on Ceph, deployable on Kubernetes via the Rook operator with dynamic CSI provisioning and optional NFS-Ganesha export for external clients, a self-managed, Kubernetes-native alternative to GCP Filestore. This guide deploys Rook + Ceph, creates a CephFS filesystem, provisions PVCs, exports over NFS, configures snapshots/backups, sets up CephX auth and tenant isolation, adds monitoring, and covers migrating off Filestore. Concept Mapping | GCP Filestore | CephFS Equivalent | Description | |---|---|---| | Filestore Instance | CephFilesystem | Distributed shared filesystem via Rook/Ceph | | Filestore NFS Export | CephNFS + NFS-Ganesha | Exposes CephFS over standard NFS | | Filestore NFSv3/v4.1 | NFS-Ganesha NFSv4 | NFS client compatibility | | Filestore Capacity Tiers | Ceph pools + StorageClasses | Performance/placement via pool settings | | Filestore Snapshots | VolumeSnapshot | CSI point-in-time snapshots | | Filestore Backups | Snapshot export to object storage | Backup via snapshots + external storage | | Filestore Multi-share | CephFS subvolumes | Isolated shared volumes via CSI | | Filestore Performance Tiers | OSD device classes | SSD/HDD OSDs with pool tuning | | Cloud Monitoring | Ceph Dashboard + Prometheus + Grafana | Metrics + visualization | Components: Rook Operator (deploys/manages Ceph), MON/MGR (quorum + monitoring), OSDs (data on block storage), MDS (CephFS metadata), Ceph CSI driver (dynamic provisioning), NFS-Ganesha (NFS export), Prometheus + Grafana (monitoring). Prerequisites: a Kubernetes cluster with 3+ worker nodes, each with a raw unformatted block storage volume attached; kubectl configured; Helm 3; basic Kubernetes storage familiarity. Install the Rook Ceph Operator $ mkdir -p ~/rook-ceph $ cd ~/rook-ceph $ kubectl get nodes Confirm all nodes are Ready . $ helm repo add rook-release https://charts.rook.io/release $ helm repo update $ helm install rook-ceph rook-release/rook-ceph --namespace rook-ceph --create-namespace --version v1.19.7 This installs the operator + CRDs. Pinned to v1.19.7 - a tested release that auto-provisions the rook-ceph.cephfs.csi.ceph.com CSI driver. If you use a newer version, confirm that driver exists before provisioning PVCs. $ kubectl get pods -n rook-ceph $ kubectl get crds | grep ceph.rook.io Deploy the Ceph Cluster $ nano ceph-cluster.yaml apiVersion: ceph.rook.io/v1 kind: CephCluster metadata: name: rook-ceph namespace: rook-ceph spec: cephVersion: image: quay.io/ceph/ceph:v19.2.3 dataDirHostPath: /var/lib/rook mon: count: 3 allowMultiplePerNode: false mgr: count: 1 allowMultiplePerNode: false dashboard: enabled: true ssl: false storage: useAllNodes: true useAllDevices: false deviceFilter: "^vd[b-z]" resources: mgr: requests: cpu: "100m" memory: "1Gi" limits: memory: "2Gi" mon: requests: cpu: "250m" memory: "512Mi" limits: memory: "1Gi" healthCheck: daemonHealth: mon: interval: 45s osd: interval: 60s deviceFilter: "^vd[b-z]" tells Rook to use only additional virtio block devices (vdb , vdc , etc.) for OSDs, excluding the primary system disk (typically vda ). Check your actual device names with lsblk on each node and adjust if needed. $ kubectl apply -f ceph-cluster.yaml $ kubectl get pods -n rook-ceph -w Deployment can take several minutes while MON/MGR/OSD pods initialize - see the Rook troubleshooting docs if pods stay pending. $ kubectl get pods -n rook-ceph | grep osd $ kubectl get cephcluster -n rook-ceph Should show PHASE: Ready , HEALTH: HEALTH_OK . Create a CephFS Filesystem $ nano ceph-filesystem.yaml apiVersion: ceph.rook.io/v1 kind: CephFilesystem metadata: name: cephfs namespace: rook-ceph spec: metadataPool: replicated: size: 3 dataPools: - replicated: size: 3 preserveFilesystemOnDelete: true metadataServer: activeCount: 1 activeStandby: true resources: requests: cpu: "100m" memory: "256Mi" limits: memory: "512Mi" 3x replication on both metadata and data pools, 1 active + 1 standby MDS for failover, filesystem data survives resource deletion. $ kubectl apply -f ceph-filesystem.yaml $ kubectl get pods -n rook-ceph -w $ kubectl get pods -n rook-ceph | grep mds $ kubectl get cephfilesystem -n rook-ceph Should show PHASE: Ready . Provision CephFS Volumes $ nano cephfs-storageclass.yaml apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: cephfs-storage provisioner: rook-ceph.cephfs.csi.ceph.com parameters: clusterID: rook-ceph fsName: cephfs pool: cephfs-data0 csi.storage.k8s.io/provisioner-secret-name: rook-csi-cephfs-provisioner csi.storage.k8s.io/provisioner-secret-namespace: rook-ceph csi.storage.k8s.io/controller-expand-secret-name: rook-csi-cephfs-provisioner csi.storage.k8s.io/controller-expand-secret-namespace: rook-ceph csi.storage.k8s.io/node-stage-secret-name: rook-csi-cephfs-node csi.storage.k8s.io/node-stage-secret-namespace: rook-ceph reclaimPolicy: Delete allowVolumeExpansion: true $ kubectl apply -f cephfs-storageclass.yaml $ kubectl get storageclass You'll see cephfs-storage alongside whatever block-storage StorageClasses your cluster ships with by default. Create a PVC and test pod: $ nano cephfs-pvc.yaml apiVersion: v1 kind: PersistentVolumeClaim metadata: name: cephfs-pvc spec: accessModes: - ReadWriteMany storageClassName: cephfs-storage resources: requests: storage: 5Gi $ kubectl apply -f cephfs-pvc.yaml $ kubectl get pvc $ nano cephfs-test-pod.yaml apiVersion: v1 kind: Pod metadata: name: cephfs-test-pod spec: containers: - name: cephfs-test-container image: busybox command: ["/bin/sh", "-c"] args: - while true; do sleep 30; done volumeMounts: - name: cephfs-storage mountPath: /mnt/cephfs volumes: - name: cephfs-storage persistentVolumeClaim: claimName: cephfs-pvc $ kubectl apply -f cephfs-test-pod.yaml $ kubectl get pod cephfs-test-pod $ kubectl exec -it cephfs-test-pod -- sh Inside the container: $ echo "CephFS volume test successful" > /mnt/cephfs/test.txt $ kubectl exec -it cephfs-test-pod -- cat /mnt/cephfs/test.txt Export CephFS via NFS $ nano ceph-nfs.yaml apiVersion: ceph.rook.io/v1 kind: CephNFS metadata: name: ceph-nfs namespace: rook-ceph spec: rados: pool: .nfs server: active: 1 resources: requests: cpu: "100m" memory: "256Mi" limits: memory: "512Mi" $ kubectl apply -f ceph-nfs.yaml $ kubectl get pods -n rook-ceph | grep nfs Deploy the toolbox to manage exports: $ kubectl apply -f https://raw.githubusercontent.com/rook/rook/master/deploy/examples/toolbox.yaml $ kubectl get pods -n rook-ceph | grep tools Create the export: $ kubectl exec -it -n rook-ceph deploy/rook-ceph-tools -- ceph nfs export create cephfs ceph-nfs /cephfs cephfs --path=/ $ kubectl exec -it -n rook-ceph deploy/rook-ceph-tools -- ceph nfs export ls ceph-nfs $ kubectl get svc -n rook-ceph | grep nfs Mount from a Linux Client The NFS service is ClusterIP by default - reachable from cluster nodes/pods only. Expose it via LoadBalancer /NodePort for external access. $ sudo mkdir -p /mnt/cephfs-nfs Install NFS client tools (Ubuntu/Debian: sudo apt install nfs-common -y ; Rocky/AlmaLinux: sudo dnf install nfs-utils -y ), then: $ sudo mount -t nfs NFS-SERVICE-IP:/cephfs /mnt/cephfs-nfs $ echo "NFS export test successful" | sudo tee /mnt/cephfs-nfs/test.txt $ cat /mnt/cephfs-nfs/test.txt Storage Pools and Performance Beyond the CephFS-backing pools, create dedicated block pools for other workloads: $ nano ceph-blockpool.yaml apiVersion: ceph.rook.io/v1 kind: CephBlockPool metadata: name: app-block-pool namespace: rook-ceph spec: failureDomain: host replicated: size: 3 parameters: compression_mode: none failureDomain: host spreads replicas across nodes; compression is off here for predictable performance. $ kubectl apply -f ceph-blockpool.yaml $ kubectl get cephblockpool -n rook-ceph $ kubectl exec -it -n rook-ceph deploy/rook-ceph-tools -- ceph status Snapshots and Backups Install CSI Snapshot CRDs $ kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/release-8.2/client/config/crd/snapshot.storage.k8s.io_volumesnapshotclasses.yaml $ kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/release-8.2/client/config/crd/snapshot.storage.k8s.io_volumesnapshotcontents.yaml $ kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/release-8.2/client/config/crd/snapshot.storage.k8s.io_volumesnapshots.yaml $ kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/release-8.2/deploy/kubernetes/snapshot-controller/rbac-snapshot-controller.yaml $ kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/release-8.2/deploy/kubernetes/snapshot-controller/setup-snapshot-controller.yaml $ kubectl get pods -n kube-system | grep snapshot Create a VolumeSnapshotClass $ nano cephfs-snapshotclass.yaml apiVersion: snapshot.storage.k8s.io/v1 kind: VolumeSnapshotClass metadata: name: cephfs-snapshotclass driver: rook-ceph.cephfs.csi.ceph.com deletionPolicy: Delete parameters: clusterID: rook-ceph csi.storage.k8s.io/snapshotter-secret-name: rook-csi-cephfs-provisioner csi.storage.k8s.io/snapshotter-secret-namespace: rook-ceph $ kubectl apply -f cephfs-snapshotclass.yaml $ kubectl get volumesnapshotclass Snapshot and Restore $ nano cephfs-snapshot.yaml apiVersion: snapshot.storage.k8s.io/v1 kind: VolumeSnapshot metadata: name: cephfs-snapshot spec: volumeSnapshotClassName: cephfs-snapshotclass source: persistentVolumeClaimName: cephfs-pvc $ kubectl apply -f cephfs-snapshot.yaml $ kubectl get volumesnapshot Restore to a new PVC (must be ≥ source size): $ nano cephfs-restore-pvc.yaml apiVersion: v1 kind: PersistentVolumeClaim metadata: name: cephfs-restore-pvc spec: accessModes: - ReadWriteMany storageClassName: cephfs-storage resources: requests: storage: 5Gi dataSource: name: cephfs-snapshot kind: VolumeSnapshot apiGroup: snapshot.storage.k8s.io $ kubectl apply -f cephfs-restore-pvc.yaml $ kubectl get pvc Scheduled Snapshots via CronJob Kubernetes does