evaluating-code-models - Evaluate code models with BigCode Evaluation Harness
Evaluates code generation models across HumanEval, MBPP, MultiPL-E, and 15+ benchmarks with pass@k metrics.
Tags
Updated: 2026-09-24Capabilities
Typical Inputs
What this skill does
- Evaluate standard code generation benchmarks
- Run multi-language code evaluation
- Benchmark instruction-tuned code models
- Compare pass@k metrics across models
- Execute evaluation safely inside Docker
Inputs
- HuggingFace model identifier or path
- Benchmark task names
- Generation configuration parameters
- Pre-generated solutions JSON file
Outputs
- JSON files containing pass@k metrics
- Saved code generation JSON files
- Model comparison tables
Requirements
- bigcode-evaluation-harness
- transformers>=4.25.1
- accelerate>=0.13.2
- datasets>=2.6.1
- Docker runtime environment
- Code execution permission