distributed-training - Distributed Training Parallelism Strategies
Provides guidance for scaling ML training across multiple GPUs and nodes with parallelism strategies
Tags
Updated: 2026-03-10Capabilities
Typical Inputs
Typical Outputs
What this skill does
- train across multiple GPUs
- select parallelism strategy
- synchronize gradients
- compress gradients
- offload to CPU
- checkpoint training state
- recover from failure
- profile GPU utilization
Inputs
- GPU cluster
- training dataset
- model architecture
- parallelism strategy
Outputs
- trained model
- checkpoint files
- training metrics
Requirements
- multiple GPUs
- NCCL backend
- torchrun
- distributed training framework