deduplication - Event Deduplication with Canonical Selection and Reputation Scoring
Remove duplicate events from multiple sources using ID-based and content-based deduplication with canonical selection
Tags
Updated: 2026-02-11Capabilities
Typical Inputs
Typical Outputs
What this skill does
- generate deduplication key
- calculate reputation score
- group similar content
- select canonical article
- calculate reduction percentage
Inputs
- article objects
- event IDs
- source results
- domain names
- published dates
Outputs
- deduplicated items
- reduction metrics
- canonical selection
- source attribution
Requirements
- TypeScript/JavaScript runtime
- hash generation capability
- multiple data sources