spark-engineer - Apache Spark Engineering and Optimization
Writes Spark jobs, optimizes queries, configures cluster settings, and handles data partitioning for big data pipelines.
Tags
Updated: 2026-09-16Capabilities
Typical Inputs
Typical Outputs
What this skill does
- Write DataFrame transformations
- Optimize Spark SQL queries
- Implement RDD pipelines
- Tune shuffle operations
- Configure executor memory
- Handle data partitioning
- Build structured streaming analytics
Inputs
- Spark job requirements
- Data schemas and files
- Cluster resource configurations
Outputs
- Spark application code
- Configuration recommendations
- Partitioning strategy explanations
- Performance analysis reports
Requirements
- Apache Spark environment
- Spark UI access