orq-build-evaluator - Build and validate binary LLM evaluators
Creates binary Pass/Fail LLM-as-a-Judge evaluators for specific failure modes and validates them against human labels using TPR/TNR.
Tags
Updated: 2026-10-05Capabilities
Typical Inputs
Typical Outputs
What this skill does
- Define failure criteria
- Design judge prompts
- Validate TPR and TNR
- Create orq.ai evaluators
- Document evaluator limitations
Inputs
- LLM pipeline details
- Failure mode definitions
- Human-labeled Pass/Fail examples
- orq.ai project access
Outputs
- Judge prompt
- Validated evaluator
- TPR and TNR metrics
- Evaluator documentation
Requirements
- orq.ai MCP or API access
- Evaluator creation permissions
- Available judge model