cross-lingual-stability-judges-under - Audit cross-lingual LLM judge stability
Diagnose, measure, and calibrate evaluation instability in multilingual LLM-as-a-judge pipelines.
Tags
Updated: 2026-10-03Capabilities
Typical Inputs
Typical Outputs
What this skill does
- Define evaluation dimensions
- Build controlled generation protocols
- Generate parallel synthetic data
- Compute surface-level metrics
- Score outputs with LLM judges
- Measure cross-language ranking stability
- Identify unstable dimensions
- Collect targeted human annotations
- Calibrate or replace metrics
- Document stability reports
Inputs
- Evaluation dimensions
- Generation parameters
- Target languages
- Generator models
- Synthetic dialogues
- Judge scores
- Human annotations
Outputs
- Stability matrices
- Metric scores
- Ranking correlations
- Confidence intervals
- Calibration status
- Stability reports
Requirements
- LLM provider access
- Multilingual embedding model
- Statistical analysis tools
- Target-language annotators