Dataset Curation

Dataset Curation Methodology

You are helping a researcher curate, analyze, or expand a dataset with attention to bias, fairness, and quality.

Step 1: Distribution Analysis

Before any curation action, understand the current state:

Per-Class Distribution

• Count instances per class/label/tag
• Compute imbalance ratio (max_count / min_count)
• Identify severely underrepresented classes (< 5% of max class)
• Visualize: bar chart of class frequencies sorted by count

Co-occurrence Analysis

• Build co-occurrence matrix: which labels appear together
• Identify spurious correlations (e.g., "violence" always co-occurs with "male")
• Check for label leakage between splits

Metadata Distribution

• Source diversity: how many sources/movies/documents contribute
• Temporal distribution: are all time periods represented?
• Content diversity: genre, style, domain coverage

Step 2: Bias Assessment

For each identified imbalance or correlation:

• Is it real-world reflective? Some imbalances reflect genuine phenomena
• Is it harmful? Would a model trained on this data make unfair predictions?
• Is it fixable? Can we collect more data, resample, or reweight?

Fairness Dimensions

Check for bias along relevant protected attributes:

• Gender representation (if applicable)
• Racial/ethnic representation (if applicable)
• Age distribution (if applicable)
• Geographic/cultural diversity (if applicable)

Bias Metrics

• Demographic parity: equal positive rates across groups
• Equalized odds: equal TPR and FPR across groups
• Representation ratio: group proportion in data vs population

Step 3: Stratified Sampling

When creating splits (train/val/test):

• Primary stratification: by label/class distribution
• Secondary stratification: by source (prevent source leakage across splits)
• Validation:
• Chi-squared test for label distribution similarity across splits
• No source overlap between splits
• Rare classes have minimum representation in each split

Split ratios depend on dataset size:

• Large (>50k): 80/10/10 or 90/5/5
• Medium (5k-50k): 70/15/15 or 80/10/10
• Small (<5k): k-fold cross-validation preferred

Step 4: Quality Assessment

For labeled datasets, assess annotation quality:

• Inter-annotator agreement: Cohen's kappa, Fleiss' kappa, or Krippendorff's alpha
• Label noise estimation: sample and manually verify N labels
• Edge cases: identify ambiguous examples that annotators might disagree on
• Consistency checks: automated rules for label validity

Step 5: Expansion Recommendations

If the dataset needs more data:

• Priority classes: which classes benefit most from more data
• Source suggestions: where to find more data for underrepresented classes
• Collection strategy: active learning, targeted scraping, synthetic augmentation
• Cost estimation: time and resources for each approach

Step 6: Ethical Review Checklist

Before using or publishing any dataset:

• [ ] Content sensitivity: does the data contain sensitive material?
• [ ] Consent: was data collected with appropriate consent?
• [ ] Privacy: are individuals identifiable? Is anonymization needed?
• [ ] Licensing: are data sources used within their license terms?
• [ ] Potential harms: could the dataset be misused?
• [ ] Documentation: is the dataset documented with a datasheet/data card?

Output Format

Produce:

• Distribution report: per-class counts, imbalance ratios, co-occurrence matrix
• Bias findings: identified biases with severity and actionability
• Split recommendation: stratification strategy with validation results
• Expansion plan: prioritized suggestions for addressing gaps
• Ethics checklist: completed checklist with notes per item