Workflow Management

Workflow Management

Version Compatibility

Reference examples assume recent stable releases of the preferred tools, especially Nextflow and the other tools listed below.

Before using code or command patterns, verify installed versions match the environment:

• Python: python -c "import <module>; print(<module>.__version__)"
• CLI: <tool> --version
• If signatures differ, inspect the installed help or API and adapt the pattern instead of retrying unchanged.

Overview

Workflow for orchestrating reproducible omics pipelines with workflow engines and clear execution provenance.

When To Use This Skill

• use when the task is to organize or run a reproducible omics pipeline
• use when Nextflow, Snakemake, CWL, or WDL style workflows are involved
• use when a one-off analysis should be turned into a repeatable pipeline

Quick Route

• If the input is raw or minimally processed data, start with validation and QC before any modeling.
• If the input is already processed, skip directly to the first workflow step that matches the user goal.
• If the user asks for a biological conclusion, always produce at least one QC or confidence artifact alongside the final result.

Progressive Disclosure

• Read references/technical_reference.md when you need deeper tool-selection rules, environment adaptation notes, or extra validation guidance.
• Keep SKILL.md as the main execution path and load the reference file only when the task or failure mode needs the extra detail.

Default Rules

• Prefer Python-first workflows unless the task explicitly requires something else.
• Keep intermediate and final outputs separated.
• Record software versions, reference builds, and key parameters when they affect interpretation.
• Favor reproducible tables and figures over one-off interactive-only outputs.

Expected Inputs

• pipeline definitions
• sample sheets
• environment descriptions

Expected Outputs

• reproducible workflow runs
• execution logs
• portable pipeline assets

Preferred Tools

• Nextflow
• Snakemake
• CWL
• WDL

Starter Pattern

nextflow run main.nf \
  --input samplesheet.csv \
  --outdir results/

Workflow

1. Define the workflow boundary

State inputs, outputs, parameters, and expected execution environment clearly.

2. Choose an engine

Use the engine already established by the project unless there is a strong reason not to.

3. Separate config from logic

Keep sample sheets, resources, and environment settings outside the core task definitions.

4. Capture provenance

Retain logs, software versions, and execution metadata for reruns.

5. Export reusable workflow assets

Save configs, manifests, and run summaries in a stable structure.

Output Artifacts

• Recommended output layout:
• results/ for final tables and serialized objects
• figures/ for plots and static visual exports
• qc/ for checks that justify downstream interpretation
• Minimum expected outputs for this skill:
• reproducible workflow runs
• execution logs
• portable pipeline assets

Quality Review

• Confirm identifiers and metadata join correctly before modeling or summarizing.
• Generate at least one QC artifact before final biological interpretation.
• Keep raw or minimally processed inputs separate from transformed outputs.
• Verify config, manifests, and sample sheets before launching a full run.
• Retain logs, versions, and the exact workflow entrypoint used for the run.

Anti-Patterns

• hardcoding sample-specific paths into pipeline logic
• mixing environment setup and workflow semantics in one opaque script
• running pipelines without recording versions and configs

Related Skills

• Sequence And Format IO
• Alignment And Mapping
• Read QC
• Database Access

Optional Supplements

• None required for the first pass.