Earth2studio Create Datasource

Create and Validate Data Source

Purpose

End-to-end workflow for implementing a new Earth2Studio data source wrapper

that connects a remote data store (S3, GCS, Azure, HTTP, HuggingFace) to

Earth2Studio's async data fetching infrastructure — from analysis through

implementation, testing, validation, and PR submission.

Prerequisites

• Earth2Studio dev environment with uv (uv run python must work)
• Git configured with fork (origin) and upstream (upstream) remotes
• Access to the target remote data store (credentials if private)
• Python 3.10+

Workspace

Use the directory containing pyproject.toml. For Harbor evals, write to

/workspace/output/ preserving paths. Never read evals/targets/.

Instructions

> Python Environment: Always use uv run python or the local .venv.

> Never use the system Python directly.

Follow every step in order.

> [CONFIRM] gates: Only Step 1 (Source Type) and Step 12 (Sanity-Check

> Plots) require explicit user approval. All other [CONFIRM] markers are

> advisory — present decisions inline and proceed without blocking.

>

> Deliverables first: Write the source file and test file (Steps 6–7)

> before extended exploration, documentation, registration, CHANGELOG, or PR

> work. Skip Steps 8–14 when the user asks for implementation only.

>

> Before you finish: Run verification commands in the repo root so results

> appear in the session log:

>

> `bash

> uv run pytest test/data/test_<source>.py -x

> make format && make lint

> `

>

> Be concise: Avoid long architecture reports; summarize decisions in a

> few sentences and move on to file writes.

>

> Hangs or User Feedback If agent becomes stuck or user provides a

> correction during this skills use, conservatively review relevant part of

> the skill and improve. Be concise.

>

> One source type per invocation. Invoke again for companion types.

Reference Files

Load these on demand during the relevant steps:

| File | Content | Load at |

|---|---|---|

| references/implementation-guide.py | Skeleton source with FILL comments | Steps 3–10 |

| references/testing-guide.py | Test skeleton with FILL comments | Step 11 |

| references/validation-guide.md | Plot templates, PR body template, Greptile handling | Steps 12–14 (optional, for templates) |

Workflow Overview

Step 0: Obtain reference → Step 1: Determine type → Step 2: Dependencies
→ Step 3: Add deps → Step 4: Create lexicon → Step 5: Update vocab/schema
→ Step 6: Create skeleton → Step 7: Implement source → Step 8: Register
→ Step 9: Documentation → Step 10: CHANGELOG → Step 11: Tests
→ Step 12: Validate & plots (user confirms) → Step 13: PR + sanity comment
→ Step 14: Greptile review

Step 0 — Obtain Remote Data Store Reference

If $ARGUMENTS is provided, use it (URL → WebFetch; file path → read).

If empty, ask:

> Please provide a URL, API documentation link, or description of the

> remote data store. This will be used to understand storage format,

> access pattern, variable inventory, temporal/spatial resolution.

Step 1 — Determine Source Type

| Protocol | Returns | Has lead_time? | Use |

|---|---|---|---|

| DataSource | xr.DataArray | No | Gridded analysis/reanalysis |

| ForecastSource | xr.DataArray | Yes | Gridded forecast |

| DataFrameSource | pd.DataFrame | No | Sparse/station obs |

| ForecastFrameSource | pd.DataFrame | Yes | Sparse forecast obs |

Key factors: gridded vs sparse → DataArray vs DataFrame; analysis vs forecast → Source vs ForecastSource.

[CONFIRM — Source Type]

Present recommended type with justification. Ask for confirmation.

Step 2 — Examine Remote Store & Propose Dependencies

Analyze: storage backend, file format, authentication, access pattern,

temporal/spatial resolution, variable inventory.

Prefer fsspec:

| Backend | Preferred | Avoid |

|---|---|---|

| AWS S3 | s3fs (core dep) | boto3 directly |

| GCS | gcsfs (core dep) | google-cloud-storage |

| Azure | adlfs | azure-storage-blob |

| HTTP | fsspec (core dep) | requests |

| HuggingFace | huggingface_hub (core dep) | custom scripts |

Only fall back to dedicated libraries when fsspec cannot access the store.

Check pyproject.toml — only propose packages not already present.

Core deps include: s3fs, gcsfs, fsspec, zarr, netCDF4, h5py,

pygrib, huggingface-hub, pandas, pyarrow.

[CONFIRM — Dependencies & Access Pattern]

Present: backend, fsspec filesystem, new packages (with license), auth method.

Step 3 — Add Dependencies

> Load references/implementation-guide.py from here through Step 10.

If new packages needed:

• uv add --extra data <package>
• uv lock
• Add optional dependency imports using OptionalDependencyFailure pattern

Step 4 — Create Lexicon Class

Create earth2studio/lexicon/<source_name>.py with:

• metaclass=LexiconType
• VOCAB: dict[str, str] mapping E2S names → remote keys
• get_item(cls, val) returning tuple[str, Callable]
• Use :: separator for structured keys

Map remote variables against E2STUDIO_VOCAB (282 entries in

earth2studio/lexicon/base.py).

[CONFIRM — Lexicon & Variable Mapping]

Present: class name, key format, full mapping table, modifiers, reference URL.

Step 5 — Update E2STUDIO_VOCAB / SCHEMA (if needed)

• New vocab: surface = descriptive abbrev; pressure = {name}{level}
• New schema fields: DataFrame sources only, check E2STUDIO_SCHEMA

[CONFIRM — Vocabulary & Schema Updates]

Skip if no updates needed.

Step 6 — Create Skeleton Data Source File

Follow canonical method ordering:

• Class constants
• SCHEMA
• __init__
• _async_init
• __call__
• fetch
• _create_tasks
• fetch_wrapper
• fetch_array

10. _validate_time

11. Helpers

12. cache property

13. available classmethod

Use async task dataclass pattern for parallel execution.

[CONFIRM — Skeleton]

Present: class name, file path, skeleton code, task dataclass.

Step 7 — Implement the Data Source & Tests

> The test file is a co-equal deliverable. Create test/data/test_<filename>.py

> alongside the source. Add test_<source>_call_mock for async sources.

Sync sources: Use prep_data_inputs/prep_forecast_inputs, direct __call__.

Async sources: See references/implementation-guide.py for required patterns:

_sync_async, managed_session, gather_with_concurrency, async_retry,

pure async I/O, try/finally cleanup. Constructor params: cache=True,

verbose=True, async_timeout=600, async_workers=16, retries=3.

DataFrame sources add time_tolerance.

Step 8 — Register the Source

• earth2studio/data/__init__.py — alphabetical import
• earth2studio/lexicon/__init__.py — alphabetical import
• Verify pyproject.toml deps

Step 9 — Update Documentation

• Add to correct RST file (datasources_analysis.rst / _forecast.rst / _dataframe.rst)
• Class docstring: Parameters, Warning (download size), Note (reference URLs), Badges (last)
• All public methods: NumPy-style docstrings

Step 10 — Update CHANGELOG.md

Add entry under the current unreleased version. See

references/implementation-guide.py REGISTRATION CHECKLIST for the format.

One line per source. Do NOT add separate lexicon entries.

Step 11 — Verify Style & Expand Tests

Run make format && make lint && make license. Load references/testing-guide.py

for test skeletons. Required tests: test_<source>_fetch (slow), _cache (slow),

_call_mock, _exceptions, _available. Target 90%+ coverage with --slow.

[CONFIRM — Tests]

Present test file, functions, coverage.

Step 12 — Validate Variables & Sanity-Check

• Validate all lexicon vars against real data (run script, do NOT commit)
• Remove variables with < 10% valid data
• Create sanity-check plot (gridded or sparse template)
• Tell user the plot path and ask for visual confirmation

[CONFIRM — Sanity-Check Plots]

User MUST visually inspect plots. Do not proceed without confirmation.

Step 13 — Branch, Commit & Open PR

• Create branch feat/data-source-<name>
• Commit (do NOT add sanity-check script/images)
• Push to fork
• gh pr create --repo NVIDIA/earth2studio
• Immediately post sanity-check validation as PR comment with:
• Variable coverage table (name, count, range, unit)
• Data validation summary (regions, storms/stations, time range)
• Key findings (physically reasonable values, conversions verified)
• Full validation script in <details> block
• Image placeholder: <!-- Drag and drop sanity-check image here -->

[CONFIRM — Ready to Submit]

Verify all steps complete before creating PR.

Step 14 — Automated Code Review

• Poll for Greptile review (5 min timeout)
• Categorize feedback (bug/style/perf/docs/suggestion/false-positive)
• Present triage table to user
• Implement accepted fixes
• Respond to PR comments
• Push

[CONFIRM — Review Triage]

User approves which comments to address.

Examples

User: Add a data source for the NOAA GFS analysis on S3
Agent: [loads skill, proceeds through Steps 0–14]

Limitations

• One source type per invocation
• Requires network access for validation (Step 12)
• SPDX license headers required in all files

Reminders

DO: uv run python, loguru.logger, alphabetical order in __init__.py/RST/CHANGELOG,

canonical method ordering, async utilities (managed_session, gather_with_concurrency,

async_retry), pure async I/O, reference URLs in docstrings, try/finally cleanup.

AVOID: asyncio.to_thread, bare tqdm.gather, xarray for loading, full file downloads.

NEVER: loop.set_default_executor(), commit secrets, commit sanity-check scripts/images.