dataset-generator - Generate and validate training datasets
Generates, normalizes, verifies, deduplicates, and exports SFT or DPO training datasets from topics, references, web research, or existing files.
Tags
Updated: 2026-09-29Capabilities
Typical Inputs
What this skill does
- Generate training datasets
- Collect source material
- Normalize dataset records
- Verify dataset records
- Deduplicate near-duplicates
- Augment dataset records
- Audit dataset files
- Export dataset formats
- Track pipeline state
Inputs
- Generation requests
- Topics or queries
- URLs
- Reference materials
- JSONL files
- CSV files
- Export schemas
- Coverage plans
- Review files
Outputs
- Training dataset files
- Canonical JSONL records
- Exported CSV files
- Exported JSONL files
- Coverage reports
- Verification results
- Deduplication status
- Pipeline progress state
- Record history
Requirements
- Python 3 environment
- Local script access
- SQLite state storage
- IDE browsing tools
- IDE search tools
- Optional semantic review support