spark-optimization - Optimize Apache Spark Job Performance
Optimize Apache Spark jobs through partitioning, caching, shuffle optimization, memory tuning, and performance debugging.
Tags
Updated: 2026-09-28Capabilities
Typical Inputs
Typical Outputs
What this skill does
- Optimize Spark job performance
- Tune memory and executors
- Implement partitioning strategies
- Optimize joins and shuffles
- Reduce data skew
- Validate optimization outcomes
Inputs
- Optimization goals and constraints
- Spark job code and configuration
- Performance symptoms and metrics
- Input data characteristics
Outputs
- Optimization recommendations
- Spark configuration changes
- Code implementation examples
- Verification steps and results