data-engineering-data-pipeline-v2 - Design scalable batch and streaming data pipelines
Provides a workflow for designing reliable, scalable, and cost-effective batch and streaming data pipelines.
Tags
Updated: 2026-09-30data engineeringdata pipelinesbatch processingstream processingETLdata qualityworkflow orchestrationdata architecture
Capabilities
What this skill does
- Assess pipeline requirements
- Select processing patterns
- Design pipeline flows
- Implement incremental loading
- Configure retry handling
- Validate data schemas
- Set up dead-letter queues
- Add observability touchpoints
- Design Kafka consumers
- Configure workflow orchestration
- Build dbt transformations
- Define data quality tests
- Select storage strategies
- Monitor pipeline performance
- Optimize pipeline costs
Inputs
- Pipeline sources
- Data volume requirements
- Latency requirements
- Target systems
- Workflow context
- Upstream support files
- Repository provenance
- Task arguments
Outputs
- Pipeline architecture recommendations
- Workflow design guidance
- Data quality checklists
- Monitoring recommendations
- Provenance-grounded review summaries