eval-harness - Build a reproducible LLM evaluation harness
Builds a reproducible harness to test LLM pipelines, score outputs, track pass/fail history, and detect regressions.
Tags
Updated: 2026-10-05Capabilities
Typical Inputs
Typical Outputs
What this skill does
- Define evaluation cases
- Create scoring rubrics
- Run pipeline evaluations
- Record pass/fail results
- Track regression history
- Analyze evaluation outcomes
Inputs
- LLM pipeline or AI feature
- Test cases
- Expected outputs
- Scoring rubrics
- Evaluation thresholds
- Reproducible execution evidence
Outputs
- Evaluation harness files
- Scored test results
- Pass/fail run history
- Regression flags
- Evaluation analysis
Requirements
- SQL session database