Arcgis To Portaljs

ArcGIS Hub → PortalJS

Overview

Migrate an entire ArcGIS Hub open-data site into a PortalJS Arc portal in one pass.

Every Hub site is machine-readable — a DCAT-US catalog at /data.json, with every dataset

backed by an ArcGIS REST FeatureService — so migration is a **harvest → export → convert →

publish → verify** pipeline that runs almost fully automated on the operator's machine, no

server-side compute. The tooling is the reusable arcgis-to-portaljs migrator: input is one

Hub URL, output is a ready-to-deploy PortalJS catalog plus a parity report.

The skill is an orchestrator: it reuses the DCAT-US harvest from portaljs-migrate, the

ogr2ogr/tippecanoe/duckdb dual-tier conversion from portaljs-add-geo, and the bulk

Git-LFS → R2 push from portaljs-migrate. Its novel parts are the FeatureService REST export

loop (paged features, not just a link) and the source-vs-derived parity report.

Prerequisites

• A scaffolded PortalJS portal whose template ships components/MapPreview.tsx and

components/GeoQuery.tsx (PR #1647 or later). Run portaljs-new-portal first if none.

• Native CLIs: GDAL (ogr2ogr, ogrinfo), tippecanoe, duckdb (with spatial),

and jq. macOS: brew install gdal tippecanoe duckdb jq; Debian/Ubuntu:

apt-get install gdal-bin duckdb jq plus tippecanoe (apt or build from source); Windows via

WSL. The skill hard-stops with the install hint if any is missing.

• Arc credentials for the Git-LFS → R2 push (the token portaljs-deploy resolves), or an OSS

self-hosted Giftless.

Instructions

The canonical, full step-by-step workflow is

.claude/commands/arcgis-to-portaljs.md

— the single source of truth. Read and follow it when executing. Summary:

• Gather input — Hub URL, portal directory, project slug, optional flags (--limit,

--only, --dry-run, --namespace-mode). Interview if missing; never dead-end.

• Check native tools (ogr2ogr, tippecanoe, duckdb + spatial, jq). Any missing →

print the per-OS install and stop.

• Validate the portal directory and confirm the geo showcase components exist.
• Harvest the Hub /data.json (reuse the portaljs-migrate DCAT-US map) and classify each

item: vector (FeatureService), table, or non-data (web map / 3D / imagery → skipped).

Under --namespace-mode owner, resolve namespaces through a publisher-normalization

table with title-prefix fallback for broken {{source}} publishers (multi-publisher

Hubs ship dirty publisher labels). Dedup near-duplicate hosted-view layers — but only

after a mandatory live record-count check on BOTH twins: equal ⇒ dedup (keep the source

layer, log the pair); different ⇒ keep both as distinct datasets. Consolidate per-year

dataset series into one year-partitioned Parquet with legacy per-year view entries. Enrich

from the AGOL item: sanitized metadata (license/description/dates), cleaned display title

(cleanTitle — raw title still drives the slug), category (item categories → meaningful

theme → keyword mapping), and a thumbnail snapshot into public/thumbnails/.

• Export each vector layer through the ArcGIS REST query API with resultOffset paging

(f=geojson, outSR=4326); fall back to keyset paging on transfer limits; accept a

customer File Geodatabase dump for very large layers.

• Convert each layer to the dual tier via the portaljs-add-geo recipe (PMTiles +

GeoParquet); tabular items to Parquet. Preserve the native-CRS original.

• Publish — bulk Git-LFS track + one push to R2 through Giftless, then append dual-tier

datasets.json entries (upsert on (namespace, slug)).

• Write arcgis-parity-report.md — record count, extent, attribute schema, and geometry

validity, source vs derived, per dataset, plus the migrated/skipped/failed accounting.

• Report the inventory, migrated datasets, R2 push, and parity summary.

Output

• Created: data/<namespace>/<slug>.pmtiles, .parquet, and the original per vector

dataset (all LFS-tracked → R2); Parquet + original per table; arcgis-parity-report.md.

• Modified: datasets.json (one dual-tier entry per vector dataset, one resource entry

per table); .gitattributes (LFS tracking).

• Verified: the parity report compares each derived artifact to the live FeatureService.
• Result: /@<namespace>/<slug> renders <MapPreview> + <GeoQuery> for each vector

dataset with no page edits; the catalog lists everything migrated.

Error Handling

| Symptom | Cause | Fix |

| --- | --- | --- |

| MISSING_INPUT | No Hub URL provided | Pass the site root (e.g. https://hub-lewisville.opendata.arcgis.com) and retry. |

| MISSING_TOOLS | ogr2ogr/tippecanoe/duckdb/jq (or duckdb spatial) absent | Print the per-OS install line and stop; re-run after installing. |

| NOT_A_PORTAL | Target dir has no datasets.json / geo components | Run portaljs-new-portal first, then re-run. |

| HARVEST_FAILED | /data.json unreachable or not DCAT-US | Confirm the site is an ArcGIS Hub and the feed loads in a browser. |

| EXPORT_FAILED | One FeatureService layer errored or hit a hard transfer cap | Logged and skipped; try keyset paging or a customer FGDB dump for that layer. |

| LFS_PUSH_FAILED | Missing/expired Arc token or unset lfs.url | Re-mint the JWT (see portaljs-deploy); confirm git config lfs.url. |

Examples

Example 1 — Migrate a City Hub (Lewisville)

/arcgis-to-portaljs https://hub-lewisville.opendata.arcgis.com slug=lewisville

Example 2 — Dry-run inventory + plan only (no writes)

/arcgis-to-portaljs https://streamwaterdata.co.uk --dry-run

Example 3 — Migrate a subset, one namespace per publisher

/arcgis-to-portaljs https://streamwaterdata.co.uk --only sewer-catchments,water-boundaries --namespace-mode owner

Resources

• Full workflow: .claude/commands/arcgis-to-portaljs.md
• REST export, classification, parity, and phasing details: references/reference.md
• Ongoing sync, parity dashboard, and cutover (Phase 3): references/sync-and-cutover.md
• Related skills: portaljs-migrate, portaljs-add-geo, portaljs-add-dataset, portaljs-deploy
• ArcGIS REST query API: <https://developers.arcgis.com/rest/services-reference/enterprise/query-feature-service-layer/> · tippecanoe: <https://github.com/felt/tippecanoe> · DuckDB spatial: <https://duckdb.org/docs/extensions/spatial>