nemo-mbridge-perf-expert-parallel-overlap - Configure MoE Expert-Parallel Overlap in Megatron-Bridge
Validate and use MoE expert-parallel communication overlap in Megatron-Bridge, including delay_wgrad_compute and flex dispatcher backends like DeepEP.
Tags
Updated: 2026-09-24Capabilities
Typical Inputs
Typical Outputs
What this skill does
- Configure expert-parallel communication overlap
- Enable delayed weight-gradient computation
- Apply flex dispatcher backends
- Validate overlap config compatibility
- Run plain-overlap isolation benchmark
- Execute communication overlap unit tests
Inputs
- Megatron-Bridge model configuration
- Communication overlap config overrides
- Slurm execution allocation
Outputs
- Updated training config
- Benchmark timing logs
- Pytest test results
Requirements
- PyTorch >= 2.6.0
- Transformer Engine >= 2.7.0
- Transformer Engine >= 2.12.0 for CUDA graph attn scope
- expert_model_parallel_size > 1
- num_moe_experts > 1
- BF16 or FP16 precision
- Ampere, Hopper, B200, B300, GB200, or GB300 GPUs for flex dispatcher