nlp - Choose tokenizers, models, and NLP metrics
Guides tokenizer selection, transformer architecture choice, sentence embeddings, and task-appropriate evaluation metrics for NLP tasks.
Tags
Updated: 2026-09-28Capabilities
Typical Inputs
Typical Outputs
What this skill does
- Choose tokenization algorithms
- Select transformer architectures
- Evaluate classification outputs
- Align NER labels
- Assess generation metrics
- Produce sentence embeddings
Inputs
- NLP task requirements
- Text samples
- Model checkpoints
- Reference texts
- Evaluation outputs
Outputs
- Tokenizer recommendations
- Architecture recommendations
- Metric recommendations
- NER evaluation guidance
- Sentence embeddings
Requirements
- Transformers-compatible model checkpoints
- Python Transformers library
- seqeval for NER evaluation
- sentence-transformers for sentence embeddings