llm-evaluation - Evaluate LLM application performance
Evaluate LLM applications using automated metrics, human assessment, LLM judges, and A/B testing.
Tags
Updated: 2026-09-28Capabilities
Typical Inputs
Typical Outputs
What this skill does
- Measure automated metrics
- Assess human evaluation dimensions
- Compare model outputs
- Run LLM-as-judge evaluations
- Analyze A/B test results
- Detect performance regressions
- Track evaluation baselines
Inputs
- LLM application outputs
- Reference answers
- Evaluation test cases
- Retrieval contexts
- Evaluation criteria
- Human ratings
- Variant scores
Outputs
- Evaluation metric scores
- Human annotation records
- LLM judge results
- Model comparison results
- A/B test statistics
- Regression findings
- Evaluation baselines
Requirements
- Python environment
- Evaluation metric libraries
- Statistical analysis libraries
- Pretrained evaluation models
- LLM API access