Slurm + Kubernetes: AI Training on HPC with ROCEv2 & IBM Spectrum Scale

// 01 — The Problem Space

Why HPC and Cloud-Native Don't Live in Isolation Anymore

Modern AI training workloads sit at an uncomfortable intersection. On one hand, HPC clusters with Slurm offer tightly coupled bare-metal scheduling, NUMA-aware placement, MPI job management, and decades of operational maturity. On the other hand, the Kubernetes ecosystem delivers container portability, GPU operator automation, operator-based distributed training (Kubeflow, PyTorchJob), and a vibrant MLOps toolchain.

The trap organizations fall into is treating these as competing philosophies. The real opportunity is to use them together — Slurm owns the outer job lifecycle, resource reservation, and policy enforcement; Kubernetes owns the inner workload containerization and AI framework orchestration. The result? You get the scheduling determinism of HPC with the flexibility of cloud-native AI stacks.

// Context This architecture is particularly relevant for LLM pre-training, multi-node NCCL-based gradient synchronization, and mixed-precision distributed fine-tuning — workloads that demand both bare-metal network latency and reproducible containerized environments.

The Stack at a Glance

Layer	Technology	Role	Why It's Here
Job Scheduler	Slurm	Outer orchestration	HPC Native
Container Orchestration	Kubernetes	Workload runtime	Cloud-Native
AI Training Framework	PyTorchJob / Kubeflow	Distributed training CRD	Ecosystem
Network Fabric	RoCEv2 (RDMA over Converged Ethernet)	GPU-to-GPU data path	Ultra-low latency
Parallel Filesystem	IBM Spectrum Scale (GPFS)	Shared model & dataset storage	POSIX Compliant
GPU Plugin	NVIDIA GPU Operator	Device management in K8s	Automation

// 02 — Infrastructure Baseline

Cluster Topology & Hardware Configuration

Before touching a single YAML file, the physical and logical topology must be right. Here's a representative production cluster design:

Node Roles

Login / Head Nodes (2×)

Slurm control daemon (slurmctld) + Kubernetes API server co-located. Users submit jobs here. These nodes see no GPU workloads directly.

Compute Nodes — GPU Workers (N×)

Slurm worker (slurmd) + Kubernetes worker (kubelet) running simultaneously. Each node exposes 8× NVIDIA H100 or AMD MI300X GPUs. Connected via RoCEv2 NICs (100GbE / 400GbE).

Storage Nodes — IBM Spectrum Scale

Dedicated NSD (Network Shared Disk) servers running GPFS daemons. Exposed over InfiniBand or high-speed Ethernet to all compute nodes. Mounted at /gpfs/projects.

Network Infrastructure

Spine-leaf Ethernet topology with DCQCN (Data Center Quantized Congestion Notification) enabled for RoCEv2 lossless fabric. Priority Flow Control (PFC) on GPU traffic class.

// Critical Slurm and Kubernetes must share the same node inventory but cannot double-book resources. The key design decision: Slurm is the authoritative scheduler. Kubernetes nodes are cordoned by default and un-cordoned by Slurm prolog scripts when a job is dispatched. This prevents both systems from independently scheduling on the same GPU.

// 03 — RoCEv2 Network Configuration

Building the Ultra-Low Latency GPU Interconnect Fabric

RoCEv2 (RDMA over Converged Ethernet v2) is the linchpin of multi-node GPU training performance. It enables GPU memory to be read/written directly across the network, bypassing the CPU — cutting latency to microseconds and enabling NCCL collective operations (AllReduce, AllGather) to approach InfiniBand speeds over standard Ethernet infrastructure.

Switch Configuration — Enabling Lossless Ethernet

          bash / switch CLI
          spine-01 # DCB & PFC configuration
        
# Enable Priority Flow Control on GPU traffic class (TC3)
dcb pfc
  priority-map 0:0 1:0 2:0 3:1 4:0 5:0 6:0 7:0   # TC3 = lossless
  pfc-cap 8

# Enable ECN for DCQCN (congestion control)
interface ethernet 1/1-32
  dcb-policy roce-policy
  no shutdown

# ECMP hashing for RoCEv2 flows (5-tuple)
ip load-sharing source-destination-port

Host-Side NIC Configuration (Mellanox / ConnectX)

          bash
          compute-node-01 # mlnx_qos + devlink
        
# Set PFC on priority 3 (RoCEv2)
mlnx_qos -i ens3f0 --pfc 0,0,0,1,0,0,0,0

# Trust DSCP for QoS marking
mlnx_qos -i ens3f0 --trust dscp

# Set RoCEv2 mode on the HCA
cma_roce_mode -d mlx5_0 -p 1 -m 2   # mode 2 = RoCEv2

# Verify RDMA device visible
ibv_devinfo | grep -E "hca_id|port_state|link_layer"
# Expected: link_layer: Ethernet  port_state: PORT_ACTIVE

# Test RDMA bandwidth between nodes
# On receiver:
ib_write_bw --report_gbits -d mlx5_0
# On sender:
ib_write_bw --report_gbits -d mlx5_0 192.168.100.11

NCCL Environment Variables for RoCEv2

          bash
          nccl_env.sh — sourced in training container
        
export NCCL_IB_DISABLE=0
export NCCL_IB_HCA=mlx5_0,mlx5_1       # bind to RoCE NICs
export NCCL_NET_GDR_LEVEL=5             # GPU Direct RDMA
export NCCL_SOCKET_IFNAME=ens3f0        # fallback TCP interface
export NCCL_DEBUG=INFO
export NCCL_ALGO=Ring                    # or Tree for large clusters
export NCCL_IB_GID_INDEX=3              # RoCEv2 GID index
export NCCL_IB_TC=106                   # DSCP → PFC class mapping

// Pro Tip Always validate your RoCEv2 fabric with perftest (ib_write_bw, ib_read_lat) before running any training job. A misconfigured PFC will cause head-of-line blocking that silently degrades AllReduce throughput by 40–60%.

// 04 — IBM Spectrum Scale Integration

Mounting GPFS as the Shared Filesystem Across Slurm & Kubernetes

IBM Spectrum Scale (formerly GPFS) provides a POSIX-compliant parallel filesystem capable of serving thousands of clients at multi-terabyte-per-second aggregate throughput. For AI training, this means checkpoint writes, dataset reads, and model artifact storage all flow through a single coherent namespace shared between Slurm compute nodes and Kubernetes pods.

Spectrum Scale on Compute Nodes

          bash
          All compute nodes — GPFS client setup
        
# Install GPFS client packages (RPM-based)
yum install -y gpfs.base gpfs.gpl gpfs.adv gpfs.crypto

# Build kernel extension
/usr/lpp/mmfs/bin/mmbuildgpl

# Add node to existing cluster
mmaddnode compute-node-01.hpc.local -N client

# Accept the cluster security keys
mmauth add compute-node-01.hpc.local -l AUTHONLY

# Mount the filesystem
mmmount gpfs0 /gpfs/projects -o relatime

# Verify mount and check performance class
mmlsmount gpfs0
mmlspolicy gpfs0

Exposing Spectrum Scale to Kubernetes Pods via PVC

The cleanest integration path is the IBM Spectrum Scale CSI driver, which provisions PersistentVolumes backed directly by GPFS filesets. Pods see a standard PVC mount at any path, but the underlying I/O hits the parallel filesystem natively.

          yaml
          spectrum-scale-sc.yaml — StorageClass
        
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: spectrum-scale-fileset
provisioner: spectrumscale.csi.ibm.com
parameters:
  volBackendFs: "gpfs0"
  filesetType:  "dependent"
  clusterId:    "2001644521943240987"
  uid:          "1000"
  gid:          "1000"
reclaimPolicy: Retain
volumeBindingMode: Immediate
allowVolumeExpansion: true

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: training-data-pvc
  namespace: ai-training
spec:
  accessModes: ["ReadWriteMany"]     # multi-node training
  storageClassName: spectrum-scale-fileset
  resources:
    requests:
      storage: 50Ti

// Design Note Use ReadWriteMany access mode for all training PVCs. This is Spectrum Scale's strength — multiple pods on different nodes can write checkpoints and read sharded datasets simultaneously without coordination overhead. InfiniBand or high-speed Ethernet between NSD servers and compute nodes is mandatory at this scale.

// 05 — Slurm ↔ Kubernetes Integration

The Bridge: Making Slurm Launch Kubernetes Workloads

This is the core architectural challenge. The solution relies on Slurm prolog/epilog scripts that orchestrate the Kubernetes API — specifically creating and destroying PyTorchJob CRDs (via Kubeflow Training Operator) when Slurm allocates and releases nodes.

The Prolog/Epilog Pattern

          bash
          /etc/slurm/prolog.d/10-kubernetes-prep.sh
        
#!/bin/bash
# Slurm Prolog — runs on each allocated compute node
# Uncordons the node from Kubernetes scheduling

NODENAME=$(hostname -s)
KUBECONFIG=/etc/kubernetes/kubelet.conf

echo "[prolog] Uncordoning K8s node: ${NODENAME}"
kubectl --kubeconfig=${KUBECONFIG} uncordon ${NODENAME}

# Label the node with the Slurm job ID for tracking
kubectl --kubeconfig=${KUBECONFIG} label node ${NODENAME} \
  slurm.job-id=${SLURM_JOB_ID} \
  slurm.partition=${SLURM_JOB_PARTITION} \
  --overwrite

exit 0

          bash
          /etc/slurm/epilog.d/10-kubernetes-cleanup.sh
        
#!/bin/bash
# Slurm Epilog — runs after job completes
# Re-cordons nodes so Kubernetes won't self-schedule on them

NODENAME=$(hostname -s)
KUBECONFIG=/etc/kubernetes/kubelet.conf

# Delete any remaining pods on this node (cleanup)
kubectl --kubeconfig=${KUBECONFIG} delete pods \
  --field-selector spec.nodeName=${NODENAME} \
  -n ai-training --force --grace-period=10

# Cordon node back from K8s scheduler
kubectl --kubeconfig=${KUBECONFIG} cordon ${NODENAME}

echo "[epilog] Node ${NODENAME} returned to Slurm-only scheduling"
exit 0

The Slurm Job Script — Launching a PyTorchJob

          bash
          submit_llm_training.sbatch
        
#!/bin/bash
#SBATCH --job-name=llm-pretrain-gpt4b
#SBATCH --partition=gpu-hpc
#SBATCH --nodes=8
#SBATCH --ntasks-per-node=1
#SBATCH --gres=gpu:8                # 8 GPUs per node = 64 total
#SBATCH --cpus-per-task=64
#SBATCH --mem=512G
#SBATCH --time=72:00:00
#SBATCH --output=/gpfs/projects/logs/llm-%j.out
#SBATCH --exclusive

JOB_NAME="llm-train-${SLURM_JOB_ID}"
NUM_NODES=${SLURM_JOB_NUM_NODES}
NAMESPACE="ai-training"

echo "Launching PyTorchJob on ${NUM_NODES} nodes, Job ID: ${SLURM_JOB_ID}"

# Generate the PyTorchJob manifest with node selectors
cat <<EOF | kubectl apply -f -
apiVersion: kubeflow.org/v1
kind: PyTorchJob
metadata:
  name: ${JOB_NAME}
  namespace: ${NAMESPACE}
  labels:
    slurm-job-id: "${SLURM_JOB_ID}"
spec:
  pytorchReplicaSpecs:
    Master:
      replicas: 1
      restartPolicy: OnFailure
      template:
        spec:
          nodeSelector:
            slurm.job-id: "${SLURM_JOB_ID}"
          containers:
          - name: trainer
            image: registry.hpc.local/ai/pytorch-rocm:2.3-rocm6.1
            resources:
              limits:
                nvidia.com/gpu: "8"
                rdma/rdma_shared_device_a: "1"
              requests:
                memory: "480Gi"
                cpu: "60"
            env:
            - name: NCCL_IB_HCA
              value: "mlx5_0,mlx5_1"
            - name: NCCL_IB_GID_INDEX
              value: "3"
            - name: NCCL_NET_GDR_LEVEL
              value: "5"
            command: ["/bin/bash", "-c"]
            args: ["torchrun --nnodes=${NUM_NODES} --nproc_per_node=8 /workspace/train.py --config /gpfs/projects/configs/gpt4b.yaml"]
            volumeMounts:
            - name: gpfs-data
              mountPath: /gpfs/projects
            - name: shm
              mountPath: /dev/shm
          volumes:
          - name: gpfs-data
            persistentVolumeClaim:
              claimName: training-data-pvc
          - name: shm
            emptyDir:
              medium: Memory
              sizeLimit: 128Gi
    Worker:
      replicas: $((NUM_NODES - 1))
      restartPolicy: OnFailure
      template:
        spec:
          nodeSelector:
            slurm.job-id: "${SLURM_JOB_ID}"
          containers:
          - name: trainer
            image: registry.hpc.local/ai/pytorch-rocm:2.3-rocm6.1
            resources:
              limits:
                nvidia.com/gpu: "8"
                rdma/rdma_shared_device_a: "1"
            command: ["/bin/bash", "-c"]
            args: ["torchrun --nnodes=${NUM_NODES} --nproc_per_node=8 /workspace/train.py --config /gpfs/projects/configs/gpt4b.yaml"]
            volumeMounts:
            - name: gpfs-data
              mountPath: /gpfs/projects
EOF

# Wait for PyTorchJob to complete
kubectl wait --for=condition=Succeeded \
  pytorchjob/${JOB_NAME} -n ${NAMESPACE} \
  --timeout=259200s

EXIT_CODE=$?
kubectl delete pytorchjob/${JOB_NAME} -n ${NAMESPACE}
exit ${EXIT_CODE}

// 06 — GPU Resource Management

GPU Operator, Device Plugins & RDMA-Aware Pod Scheduling

The Kubernetes GPU Operator automates the entire GPU software stack — drivers, container toolkit, device plugin, and DCGM exporter. But for RoCEv2-backed NCCL training, we also need RDMA device exposure inside pods via the RDMA Shared Device Plugin.

          bash
          GPU Operator install via Helm
        
# Add NVIDIA Helm repo
helm repo add nvidia https://helm.ngc.nvidia.com/nvidia
helm repo update

# Install GPU Operator with RDMA and MIG support
helm install gpu-operator nvidia/gpu-operator \
  --namespace gpu-operator \
  --create-namespace \
  --set driver.enabled=true \
  --set mig.strategy=mixed \
  --set toolkit.enabled=true \
  --set devicePlugin.enabled=true \
  --set rdma.enabled=true \
  --set rdma.useHostMofed=true \
  --set dcgmExporter.enabled=true   # Prometheus metrics

          yaml
          rdma-device-plugin-config.yaml
        
apiVersion: v1
kind: ConfigMap
metadata:
  name: rdma-devices
  namespace: kube-system
data:
  config.json: |
    {
      "periodicUpdateInterval": 300,
      "sharedDevices": [
        {
          "name": "rdma_shared_device_a",
          "vendors": ["15b3"],           // Mellanox
          "deviceIDs": ["101b"],         // ConnectX-6
          "rdmaHcaMax": 64
        }
      ]
    }

Topology-Aware Pod Placement

For best GPU-to-GPU RDMA performance, pods must land on nodes where the GPU and the RoCEv2 NIC share the same PCIe root complex (same NUMA socket). Use Topology Manager and CPU Manager in Kubernetes to enforce this.

          yaml
          /var/lib/kubelet/config.yaml — kubelet topology policy
        
topologyManagerPolicy: single-numa-node
cpuManagerPolicy: static
reservedSystemCPUs: "0-3"
memoryManagerPolicy: Static
reservedMemory:
  - numaNode: 0
    limits:
      memory: 4Gi

// 07 — Observed Performance Results

What This Architecture Delivers in Practice

After running this stack in production across several large-scale LLM pre-training runs, here are representative numbers from an 8-node × 8× H100 SXM5 cluster:

~180

GB/s inter-node bandwidth (RoCEv2 AllReduce, 100GbE)

<3μs

RDMA read latency (node-to-node, same rack)

94%

GPU utilization sustained over 72h training run

40TB/s

Aggregate Spectrum Scale filesystem throughput (peak)

// Takeaway The Slurm-first scheduling model is what made the 94% GPU utilization possible. By giving Slurm exclusive node ownership before Kubernetes ever sees the node, we eliminated the "noisy neighbor" problem that plagues shared Kubernetes clusters. Jobs got exactly what they asked for, every time.

// 08 — Operational Considerations

What to Watch Out For in Production

Checkpoint Strategy on Spectrum Scale

Configure PyTorch DCP (Distributed Checkpoint) to write to /gpfs/projects/checkpoints/{job_id}/. Spectrum Scale handles concurrent multi-writer checkpoints efficiently via its metadata caching architecture. Set mmchconfig maxFilesToCache=8192000 on NSD servers for large training jobs that open thousands of files concurrently.

Slurm Resource Binding vs Kubernetes Requests

A common pitfall: Slurm's --gres=gpu:8 reserves the GPU at the hypervisor level (cgroups), but Kubernetes GPU requests must match exactly. If Kubernetes allocates 7 GPUs on a node that Slurm reserved 8 for, the 8th GPU is orphaned. Enforce matching by templating the PyTorchJob manifest from Slurm environment variables as shown in Section 05.

RoCEv2 Packet Loss Detection

NCCL will hang silently on even 0.01% packet loss if ECN/PFC is misconfigured. Install DCGM and monitor DCGM_FI_DEV_NVLINK_BANDWIDTH_TOTAL alongside switch counters. Set up alerting on cnp_sent (Congestion Notification Packets) — a spike here means your fabric is congested and training throughput will degrade.

IBM Spectrum Scale Quota Management

Use GPFS filesets and quotas per project namespace: mmcrfileset gpfs0 project-nlp --inode-space new then link and set quota. This prevents any single training job from consuming the entire filesystem and creates natural accounting boundaries aligned with your Slurm partitions and Kubernetes namespaces.

// Security Note The kubelet config used in prolog/epilog scripts should be a dedicated service account with RBAC limited to: get/list/update nodes, create/delete/get pytorchjobs, delete pods in the ai-training namespace only. Never use admin kubeconfig in Slurm scripts.

// 09 — Closing Thoughts

The Future is Hybrid, Not Either/Or

The instinct to standardize on one orchestrator — either Slurm or Kubernetes — is understandable but increasingly counterproductive. The HPC world has decades of scheduler maturity, accounting rigor, and bare-metal performance that cloud-native tooling is only beginning to replicate. Conversely, the AI ecosystem — PyTorch, Kubeflow, Ray, Volcano, MLflow — is built natively for Kubernetes.

The architecture described here is not a hack or a workaround. It is an intentional composition of two scheduling layers, each operating at the level of abstraction it's best suited for. Slurm thinks in nodes and partitions. Kubernetes thinks in pods and operators. RoCEv2 speaks in microseconds. Spectrum Scale speaks in petabytes. Put them together with care, and you get an AI training platform that matches or exceeds what the large cloud providers offer — on your own hardware, under your own SLAs.

The teams building foundation models at national labs, automotive OEMs, and financial institutions are already running this stack. The tooling has matured. The patterns are proven. The only remaining barrier is organizational — and that's a different blog post entirely.

#HPC #Kubernetes #Slurm #AIInfrastructure #RoCEv2 #RDMA #IBMSpectrumScale #GPFS #DistributedTraining #PyTorch #Kubeflow #NCCL #GPUClusters #MLOps #LLM

Found this useful?

If you're working on HPC + Kubernetes integration, GPU cluster optimization, or distributed AI training infrastructure, let's connect and exchange notes.

Connect on LinkedIn →