OpenShift persistent storage on a Fibre Channel SAN: static PV/PVC workflow

Dynamic provisioning through a vendor CSI driver is the usual answer for enterprise storage on OpenShift. Sometimes it is not available yet, is not approved, or the organisation wants every volume to be deliberately created and owned. In that case, a manual, static workflow is a legitimate, controlled model: storage presents a LUN, platform validates it, and the application team receives a named PV and PVC.

This article describes that workflow for Fibre Channel LUNs presented to OpenShift worker nodes, including its limits.

This procedure was validated for the OpenShift 4 static local-PV pattern described here, not as a generic shared Fibre Channel design. Revalidate Local Storage Operator support, multipath configuration, RHCOS persistence, SELinux behavior and recovery steps before a version-family, storage-firmware, topology or node-pool change.

The decision, written down#

1. LUN request template#

Make requesters fill in the same fields every time.

The storage team replies with the LUN ID and WWN. Do not create PVCs before that confirmation.

2. Check visibility on the worker#

These are privileged host operations, not commands to run in an ordinary application pod or on an administrator workstation. Work on the designated worker through an authorized oc debug node/... session. Stop if the host identity or WWN does not match the approved change record.

oc get nodes -o wide
oc debug node/wk01
chroot /host

mpath_device=REPLACE_WITH_VERIFIED_MULTIPATH_DEVICE_NAME
lsblk
multipath -ll
lsscsi || true
udevadm info --query=all --name="/dev/mapper/$mpath_device" | grep -E 'ID_SERIAL|DM_UUID|DM_NAME|ID_WWN'
blkid "/dev/mapper/$mpath_device" || true
exit; exit

Match the WWN against the LUN ID the storage team gave you. Confirm all expected paths are active and the device is a mpath device, not a single path (/dev/sdX).

3. Prepare the filesystem and mount#

Formatting is a one-time storage-owner action. Before it, prove from the array mapping and host output that the WWN is the newly allocated, empty LUN; check that it has no filesystem, signatures, partitions or active mounts; and obtain the required destructive-change approval. The following is deliberately pseudocode rather than a copy-and-run command:

# On the designated node, after the destructive checks and approval:
# Run the reviewed filesystem, mount-point and persistent-mount procedure here.
# No generic formatting command is provided because selecting the wrong LUN destroys data.

RHCOS nodes are managed by the Machine Config Operator, so do not rely on hand-edited files for persistence. Use a supported local-storage workflow for your OpenShift release, preferably the Local Storage Operator when applicable. If an approved MachineConfig/systemd mount design is used, target only a dedicated MachineConfigPool whose nodes all satisfy the storage assumptions; a worker-wide unit can make unrelated nodes fail boot or repeatedly attempt to mount a LUN they cannot see. Define ownership and SELinux requirements explicitly and test them with the workload's run-as UID/GID instead of making the host path broadly writable.

Fail closed when the SAN mount is absent#

A plain directory at /mnt/san/billing-api/data is dangerous: if the SAN filesystem disappears, kubelet can write application data into the node's root filesystem. The production design must prevent scheduling and writes unless the expected multipath-backed filesystem is mounted.

• Make the persistent mount unit depend on the stable UUID or WWID, require _netdev/device readiness as appropriate, and order the validated mount before kubelet through a release-supported MachineConfig/systemd design.
• Use a dedicated mount point, not a directory that also contains local node data. Record the expected source identity, filesystem UUID, type and mount options.
• Add a privileged host-level guard managed through MachineConfig or approved node automation. It must verify findmnt --mountpoint, compare the mounted source/UUID to inventory, and keep the dedicated node tainted or unschedulable when the check fails. Remove the protective taint only after the expected mount is verified.
• Alert on mount loss, multipath degradation, root-filesystem growth and pods using the PV while the guard is unhealthy. Test cable/path loss and a full mount loss in a change window.
• Never let an init container merely create or test-write the host directory; directory existence does not prove the SAN is mounted.

Review-first verification on the designated host:

mount_path=/mnt/san/billing-api/data
expected_uuid=REPLACE_WITH_RECORDED_FILESYSTEM_UUID
findmnt --mountpoint "$mount_path" -o SOURCE,FSTYPE,OPTIONS,TARGET
test "$(findmnt -n -o UUID --target "$mount_path")" = "$expected_uuid"
findmnt -n -o SOURCE --target /
findmnt -n -o SOURCE --target "$mount_path"

The last command must show different backing sources for / and mount_path. If any check fails, cordon and drain according to the workload disruption policy, stop the workload, and restore the expected mount; do not create the PV or uncordon the node.

4. StorageClass, PV and PVC#

A local-path PV needs a StorageClass that does not provision and binds late:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: manual-san-fc
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer

Then the volume and claim:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv-billing-api-data
spec:
  capacity:
    storage: 200Gi
  volumeMode: Filesystem
  accessModes:
  - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: manual-san-fc
  local:
    path: /mnt/san/billing-api/data
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - wk01
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pvc-billing-api-data
  namespace: billing-prod
spec:
  accessModes:
  - ReadWriteOnce
  volumeMode: Filesystem
  resources:
    requests:
      storage: 200Gi
  storageClassName: manual-san-fc
  volumeName: pv-billing-api-data

Name PVs by application and purpose. In six months, pv-billing-api-data tells an operator what it is; pv-0007 does not.

5. Validate#

oc get pv,pvc -A
oc describe pvc pvc-billing-api-data -n billing-prod

After the application is deployed, confirm it mounts the claim, writes a test file, and survives a pod restart. Record the output as evidence.

Also test the fail-closed control: stop the workload, use the approved storage-loss simulation, and confirm the node becomes unavailable for this workload before the SAN path can fall back to the root filesystem. Restore the mount, verify WWID/UUID and multipath health, remove the protective taint under change control, and then restart the workload. Roll back the storage change if identity, SELinux access, mount persistence or the guard test fails.

Retained-volume lifecycle#

Retain deliberately leaves data and the PV for an administrator after PVC deletion. A Released PV must never be rebound casually.

1. Stop the workload and verify no pod uses the claim; capture PV, PVC, namespace, WWN, UUID and backup evidence.
2. Delete the PVC only under approved change control. Confirm the PV becomes Released and keep the LUN mapped, mounted read-only where practical, and protected from reuse.
3. For recovery or reassignment, validate the data owner and backup, then follow the selected release's documented retained-PV reuse procedure. Review and remove only the old claimRef when intentional; never change it while the old claim or workload exists.
4. For decommissioning, obtain data-owner and storage-owner approval, unmount, remove node persistence and PV objects, unmap the exact WWN, and sanitize or destroy data under the retention policy. Deleting a PV object does not erase the LUN.
5. Record every transition. Roll back before unmapping if any identity or ownership check is ambiguous; after sanitization there is no data rollback.

Know the trade-offs#

• Single-node ownership. This manifest is a Kubernetes local PV, not a shared FC volume. Its node affinity permanently binds it to wk01; presenting the same LUN to more workers does not make this PV fail over and can allow unsafe concurrent filesystem access.
• Failure recovery. If the designated node fails, the pod remains unavailable. A documented recovery must fence or prove the old node is off, unmount/detach the LUN, map and mount it on the replacement, and create or update storage objects using the release-supported procedure. Never mount a single-node XFS filesystem from two nodes.
• No automated expansion or snapshots. Those come with a CSI driver.
• Operational load. Every volume is a small ticket. That is the point of the model, and also its cost.

When the volume count or the recovery requirement grows, evaluate your storage vendor's CSI driver for Fibre Channel. The PV and PVC naming and ownership discipline above carries over directly.

Handover record per volume#

LUN ID, WWN, filesystem UUID, designated worker, visibility output, PV/PVC YAML, mount and guard definitions, mount path, service and storage owners, backup requirement, retained-volume state, and tested mount-loss and node-loss recovery procedures.

Examples use documentation-only names. Confirm behaviour for your storage array and OpenShift version.

Planning storage for OpenShift or a migration from VMs? See Kubernetes and OpenShift and Backup and Recovery, or start a project.