Algo Rank Wilson Agent Skill

Wilson Score Ranking

Overview

Wilson Score interval provides a lower confidence bound on the true proportion of positive ratings. Unlike simple averages, it penalizes items with few ratings, preventing a 5/5 review item (1 review) from outranking a 4.8/5 item (1000 reviews). Computes in O(1) per item.

When to Use

Trigger conditions:

• Ranking items by user ratings when review counts vary widely
• Building "top rated" or "best of" lists that are fair to well-reviewed items
• Sorting binary feedback (upvote/downvote) with confidence

When NOT to use:

• For continuous scores (use Bayesian average instead)
• When comparing items with similar sample sizes (simple average suffices)

Algorithm

IRON LAW: Never Rank by Simple Average When Sample Sizes Differ
A 5.0 average from 1 review is NOT better than 4.8 from 1000 reviews.
Wilson Score lower bound accounts for sample uncertainty:
Items with few ratings get a LOWER bound, properly reflecting our
uncertainty about their true quality.

Phase 1: Input Validation

Collect per item: number of positive ratings (p), total ratings (n). For star ratings, convert to binary (e.g., 4-5 stars = positive).

Gate: n > 0 for all items, confidence level chosen (typically 95%, z=1.96).

Phase 2: Core Algorithm

• Compute observed proportion: p̂ = positive / total
• Wilson lower bound: (p̂ + z²/2n - z × √(p̂(1-p̂)/n + z²/4n²)) / (1 + z²/n)
• Rank by Wilson lower bound descending (conservative estimate of true quality)

Phase 3: Verification

Check: items with many positive reviews rank above items with few reviews and same proportion. Items with very few reviews are appropriately penalized.

Gate: Ranking intuitively correct on manual inspection.

Phase 4: Output

Return ranked items with scores and confidence intervals.

Output Format

{
  "rankings": [{"item": "Product_A", "wilson_lower": 0.89, "positive": 950, "total": 1000, "proportion": 0.95}],
  "metadata": {"confidence": 0.95, "z": 1.96, "items_ranked": 500}
}

Examples

Sample I/O

Input: Item A: 1 positive / 1 total (100%). Item B: 950 positive / 1000 total (95%).

Expected: B ranks higher. Wilson lower: A ≈ 0.05, B ≈ 0.94. The single review gives almost no confidence.

Edge Cases

| Input | Expected | Why |

|-------|----------|-----|

| 0 reviews | Cannot rank | n=0, undefined. Exclude or assign minimum |

| 0 positive, 100 total | Very low score | Genuinely bad item, high confidence |

| 1M positive, 1M total | Lower bound ≈ 1.0 | Massive sample, high confidence in 100% |

Gotchas

• Binary conversion: For 5-star ratings, the positive/negative threshold matters. 4+ stars as positive? 3+ stars? Different thresholds produce different rankings.
• Not for continuous data: Wilson Score is for proportions (binary outcomes). For continuous ratings, use Bayesian average with a prior.
• Cold start: New items with zero reviews can't be ranked. Use a minimum review threshold or Bayesian smoothing.
• Confidence level choice: Higher confidence (99%) penalizes small samples more aggressively. 95% is standard but tune for your use case.
• Sorting by lower bound is conservative: This approach favors well-known items. For discovery/exploration, consider also boosting items with high upper bounds (potential hidden gems).

Scripts

| Script | Description | Usage |

|--------|-------------|-------|

| scripts/wilson_score.py | Compute Wilson score interval and rank items | python scripts/wilson_score.py --help |

Run python scripts/wilson_score.py --verify to execute built-in sanity tests.

References

• For Bayesian average alternative, see references/bayesian-average.md
• For Reddit's ranking algorithm (Wilson-based), see references/reddit-ranking.md