mcpbeat

Dedupe Rank

willoscar/dedupe-rank

| Use when a broad paper candidate pool needs deterministic deduplication and a stable core set.

6k tokens
context cost
the whole folder, loaded on every use
5
files
ships runnable scripts
0
copies elsewhere
how many repositories repackaged it
496
stars on the repo
on the repository, not the skill itself

Install

one command, takes just this skill from the repository 
npx skills add https://github.com/WILLOSCAR/research-units-pipeline-skills --skill dedupe-rank

What comes with it

21 837 bytes besides the instruction
assets/domain_packs/embodied_ai.json
assets/domain_packs/llm_agents.json
references/domain_pack_overview.md
scripts/run.py

The instruction itself

7 sections, as written by the author

Dedupe + Rank

Turns a raw candidate pool into a deduped pool and a stable core set.

Input

  • papers/papers_raw.jsonl

Outputs

  • papers/papers_dedup.jsonl
  • papers/core_set.csv

Script boundary

scripts/run.py should own only:

  • title/year deduplication
  • deterministic ranking
  • stable paper_id generation

Use shared domain packs or pipeline contract metadata for topic-specific or product-specific behavior.

Contract-driven behavior

The script should prefer pipeline contract metadata over profile-name branching.

Current important field:

  • quality_contract.candidate_pool_policy.keep_full_deduped_pool
  • quality_contract.candidate_pool_policy.include_domain_pins
  • quality_contract.candidate_pool_policy.minimum_domain_surveys
  • quality_contract.candidate_pool_policy.survey_title_bonus

If true, the script keeps the full deduped pool in papers/core_set.csv unless the user explicitly overrides core size.

Focused briefing Workflows can disable unconditional domain pins, reserve a

smaller survey floor, and remove the generic survey-title bonus without

changing broad Survey defaults.

Acceptance

  • deduped JSONL exists
  • core-set CSV exists
  • reruns are stable for the same inputs

Non-goals

  • retrieval
  • screening
  • manual topic authoring inside the script

How to use it

Copy the folder

Take willoscar/dedupe-rank from the repository into ~/.claude/skills for personal use, or into .claude/skills inside a project.

Check the name does not clash

The agent identifies a skill by the name field in its header. Two skills with the same name cannot sit side by side — one of them will be ignored.