streaming-data-engineer - Build and operate production streaming data pipelines
Provides hands-on guidance for Kafka, Flink, and Spark streaming pipelines, including windows, exactly-once processing, backpressure, dead-letter queues, and monitoring.
Tags
Updated: 2026-10-03Capabilities
Typical Inputs
Typical Outputs
What this skill does
- Tune Kafka producers
- Configure Kafka consumers
- Implement Flink windows
- Build Spark streaming jobs
- Manage exactly-once processing
- Handle stream backpressure
- Publish dead-letter messages
- Evolve event schemas
- Monitor streaming jobs
Inputs
- Kafka broker configuration
- Kafka topic names
- Event schemas
- Stream processing requirements
- Checkpoint locations
- Output destinations
Outputs
- Processed stream events
- Windowed aggregation results
- Dead-letter queue messages
- Streaming job status
Requirements
- Kafka environment
- Apache Flink environment
- Apache Spark Structured Streaming
- Read and write topic permissions
- Checkpoint storage access