Modern AI training workloads sit at an uncomfortable intersection. On one hand, HPC clusters running Slurm offer tightly coupled bare-metal scheduling, NUMA-aware placement, MPI job management, and decades of operational maturity. On the other hand, the Kubernetes ecosystem delivers container portability, GPU operator automation, operator-based distributed training (Kubeflow, PyTorchJob), and a vibrant MLOps toolchain.
The trap organizations fall into is treating these as competing philosophies. The real opportunity is to use them together — Slurm owns the outer job lifecycle, resource reservation, and policy enforcement; Kubernetes owns the inner workload containerization and AI framework orchestration.
Five pillars, one stack
The stack is built around five pillars: Slurm as the authoritative outer scheduler, Kubernetes for containerized workload management, RoCEv2 for ultra-low-latency GPU interconnect, IBM Spectrum Scale (GPFS) as the shared parallel filesystem, and the NVIDIA GPU Operator for device lifecycle automation.
| Layer | Component | Role |
|---|---|---|
| Scheduling | Slurm | Outer job lifecycle, reservation, policy enforcement |
| Orchestration | Kubernetes | Container placement, PyTorchJob / Kubeflow operators |
| Interconnect | RoCEv2 | RDMA fabric for NCCL collectives, GPU-direct networking |
| Storage | IBM Spectrum Scale (GPFS) | Shared parallel filesystem, checkpoint & dataset I/O |
| Device Mgmt | NVIDIA GPU Operator | Driver, toolkit, and device plugin lifecycle |
RoCEv2 configuration
RoCEv2 enables GPU memory to be read and written directly across the network, bypassing the CPU — cutting latency to microseconds and enabling NCCL collective operations (AllReduce, AllGather) to approach InfiniBand speeds over standard Ethernet infrastructure.
Key configuration: enable PFC on traffic class 3, set DCQCN for congestion control, and configure NCCL_IB_GID_INDEX=3 for RoCEv2 GID selection.
ib_write_bw and ib_read_lat before running any training job. A misconfigured PFC causes head-of-line blocking that can silently degrade AllReduce throughput by 40–60%.The Slurm ↔ Kubernetes bridge
The integration relies on Slurm prolog/epilog scripts that orchestrate the Kubernetes API — creating and destroying PyTorchJob CRDs as Slurm allocates and releases nodes. Slurm remains the authoritative scheduler; nodes are cordoned by default and un-cordoned by prolog scripts only when a job is actually dispatched.
\# prolog.sh (excerpt)
kubectl uncordon "$SLURM_NODELIST"
kubectl apply -f pytorchjob-${SLURM_JOB_ID}.yaml
\# epilog.sh (excerpt)
kubectl delete pytorchjob training-${SLURM_JOB_ID}
kubectl cordon "$SLURM_NODELIST"
IBM Spectrum Scale integration
GPFS provides a POSIX-compliant parallel filesystem serving thousands of clients at multi-terabyte-per-second aggregate throughput. The IBM Spectrum Scale CSI driver provisions PersistentVolumes backed directly by GPFS filesets, with ReadWriteMany access for concurrent multi-node training — critical for checkpoint writes and shared dataset reads across a distributed job.
Measured on an 8-node × 8×H100 cluster
Running this stack across 8-node × 8×H100 clusters over sustained 72-hour training runs: