Portaljs Check Data Quality

PortalJS — Check Data Quality

Overview

Run a read-only quality audit of one CSV or TSV file, local or remote, and return a

structured JSON report. The audit profiles every column — null/blank counts, inferred

value types, numeric ranges, likely year/date fields — and flags duplicate rows,

duplicate values in identifier-like columns, ambiguous overlapping year columns (e.g.

calendar year vs fiscal year), and mixed-type columns. It never edits the source

file, datasets.json, or any other project file; it only reads the target file (a

remote URL is downloaded to a temp file that is deleted before the run ends) and

prints a report. Use it before publishing a dataset with portaljs-add-dataset, or to

diagnose why a showcase renders wrong.

Prerequisites

• python3 on PATH — the audit logic runs as an embedded Python script; nothing is

installed.

• One CSV or TSV file, given as a local path or an http/https URL. Only one file

per run.

Instructions

The canonical, full step-by-step workflow is

.claude/commands/portaljs-check-data-quality.md —

the single source of truth. Read and follow it when executing. Summary:

• Gather input — the file path or URL to audit. If missing, ask for it; never dead-end.
• Resolve the source: if it's an http/https URL, download it to a temp file first;

otherwise use the local path as given.

• Validate the extension is .csv or .tsv. If not, or the file is missing, or the

header row is empty, stop and surface the error JSON as-is — do not guess a fix.

• Profile every column: null/blank counts, distinct values, sample values, inferred

per-value type (boolean/integer/float/date/string), numeric min/max, and year

range for columns whose name looks year-like.

• Derive findings from the profiles — duplicate rows, missing-value ratios, invalid

year values, mixed types, suspect negative values, duplicate identifier values, and

ambiguous overlapping year columns — each tagged critical, warning, or info.

• Assemble the JSON report (status, file metadata, findings, recommendations,

column_profiles), print it, and clean up the temp file if one was created.

• Relay the report to the user as-is; do not modify the source file, datasets.json,

or any other project file based on the findings — that's a separate, explicit step.

Output

A single JSON object printed to stdout:

• status — ok, warning, or critical.
• file, file_name, source_type (local or url), row_count, column_count.
• findings — structured issues, most severe first.
• recommendations — de-duplicated suggested next steps.
• column_profiles — per-column summary (nulls, blanks, distinct count, sample

values, inferred types, numeric/year ranges).

No files are created or modified. A remote URL's temp download is removed on exit,

success or failure alike.

Error Handling

| Symptom | Cause | Fix |

| --- | --- | --- |

| "File ... is not available." | Local path is wrong, or the URL download failed | Verify the path or URL is reachable and retry. |

| "Only CSV and TSV files are supported right now." | File extension isn't .csv/.tsv | Convert the file, or point to its tabular source instead. |

| "... does not contain tabular headers." | File is empty or the header row is malformed | Open the file and confirm it has a valid, non-empty header line. |

| Command hangs on a URL | Remote host is slow or blocks non-browser requests | Download the file manually and audit the local copy instead. |

| python3: command not found | Python 3 isn't installed or not on PATH | Install Python 3, or run the audit where it's available. |

| Report looks truncated in the terminal | Large report wrapped/paginated by the shell | Redirect to a file (> report.json) and open it separately. |

Examples

Example 1 — Audit a local CSV before publishing

/portaljs-check-data-quality ./public/data/trash.csv

Example 2 — Audit a remote CSV over HTTPS

/portaljs-check-data-quality https://example.com/trash.csv

Example 3 — Audit a TSV and save the report for review

bash scripts/check-data-quality.sh ./data/emissions.tsv > /tmp/emissions-quality.json

Example 4 — Read a critical status report

{
  "status": "critical",
  "findings": [
    { "severity": "critical", "check": "duplicate_rows", "message": "42 duplicate rows found." }
  ],
  "recommendations": ["Review and deduplicate repeated rows if they are not intentional."]
}

Fix the flagged rows/columns, then re-run the audit before publishing.

Resources

• Full workflow: .claude/commands/portaljs-check-data-quality.md
• Detailed check catalog and troubleshooting: references/reference.md
• Related skills: portaljs-add-dataset, portaljs-define-schema
• Python csv module (parsing behavior this audit relies on): <https://docs.python.org/3/library/csv.html>