experiments-index - Incrementally append records to FAISS and HuggingFace store
Appends campaign result records to an existing FAISS index and HuggingFace dataset under an fcntl file lock.
Tags
Updated: 2026-09-18Capabilities
Typical Inputs
What this skill does
- Acquire exclusive fcntl file lock
- Embed text records with encoder
- Normalize embeddings using L2 norm
- Append vectors to FAISS index
- Concatenate records into Arrow dataset
- Save updated store to disk
Inputs
- Campaign result records
- Target FAISS index file path
- Target HuggingFace dataset directory path
- Sentence-embedding encoder model
Outputs
- Updated FAISS index file
- Updated HuggingFace Arrow dataset directory
- JSON summary of appended count and total index size
Requirements
- Python environment with faiss-cpu or faiss
- Python HuggingFace datasets package
- Sentence-embedding encoder model
- File system supporting fcntl locks
