embedding-analysis - Dataset Embedding Analysis
Compute and analyze embeddings for dataset quality, distribution comparison, deduplication, and diversity measurement
Tags
Updated: 2026-06-30Capabilities
Typical Inputs
Typical Outputs
What this skill does
- generate embeddings
- semantic deduplication
- compare distributions
- measure diversity
- score quality
- filter examples
- compute similarity
- calculate perplexity
Inputs
- text dataset
- image dataset
- embedding model
- reference model
- train/val/test splits
- gold reference set
Outputs
- embedding vectors
- deduplication mask
- distribution metrics
- quality scores
- diversity metrics
- perplexity scores
- filtered dataset
Requirements
- Python 3.8+
- sentence-transformers
- scikit-learn
- numpy
- umap-learn
- GPU support (optional)