Physical AI People Attribute Search Agent Skill

Physical AI People Attribute Search Workflow Orchestrator

Default workflow skill for PAS execution on OSMO. It owns flow selection,

preflight, submit-time interpolation, monitoring, and output retrieval.

Purpose

Run the PAS image augmentation and auto-labeling pipeline safely and

reproducibly from preflight to output download.

The PAS pipeline augments existing person-crop datasets by generating

controlled clothing/appearance variations (image-domain) and synonymous

attribute captions (text-domain). It uses the paidf-augmentation container

for image-edit augmentation with MCQ verification, and the

paidf-auto-labeling container for person-attribute captioning.

Do NOT use this skill for container-internal tuning-only questions.

Prerequisites

Confirm these before running preflight or any submit. Missing required secrets

surface as USER_INPUT_REQUIRED: from scripts/preflight_credentials.sh.

| Requirement | How it is satisfied | Used for |

|---|---|---|

| NGC API key (optional) | NGC_API_KEY, NGC_CLI_API_KEY, or compatible nvapi-* token | Optional for nvcr_io credential refresh; default PAS image refs are public |

| Hugging Face token | HF_TOKEN (or HUGGING_FACE_HUB_TOKEN), or a cached token at ~/.cache/huggingface/token | Creates the OSMO hf_token credential |

| OSMO CLI access | osmo on PATH, logged in, with a default profile and a registered DATA credential profile matching storage_url | Submitting/monitoring workflows and listing/downloading objects |

| GPU pool | At least one ONLINE pool in osmo pool list --mode free | Scheduling setup + worker tasks |

| Image Edit endpoint | In-cluster NIM qwen-image-edit-2511 (reused if healthy, else deployed via the NIM operator); external opt-in via image_edit_url | Image-domain augmentation |

| VLM endpoint | In-cluster NIM qwen3-vl (shared with VDA); external opt-in via vlm_url | MCQ verification and person-attribute captioning |

| LLM endpoint | In-cluster NIM qwen25-14b (shared with VDA); external opt-in via llm_url | MCQ question generation |

Instructions

• Select the workflow (e2e, augmentation, auto_labeling) from user intent.
• Provide a tentative execution-time overview before starting run actions.
• Run preflight and readiness checks before submit.
• Derive submit-time values from the active dataset backend (never guess

storage_url).

• Submit the workflow with explicit interpolation values and monitor to completion.
• Retrieve outputs and summarize task outcomes.

Use run_script(...) for script execution. Canonical examples:

run_script("bash scripts/preflight_credentials.sh --workflow assets/configs/osmo/e2e.yaml")

Available Scripts

Use script-level --help for exact arguments.

| Script | Role |

|---|---|

| scripts/preflight_credentials.sh | Secrets/control-plane preflight and workflow image access checks |

| scripts/augmentation_worker.sh | Image-edit augmentation worker (preprocess, config gen, augment, post-process) |

| scripts/auto_labeling_worker.sh | Person-attribute captioning worker |

| scripts/endpoint_common.sh | Shared endpoint health/auth helpers |

Supported Flows

| Flow | OSMO YAML | Group sequence | Typical use |

|---|---|---|---|

| e2e | assets/configs/osmo/e2e.yaml | setup -> augmentation -> auto_labeling | Full pipeline: augment person crops then generate captions |

| augmentation | assets/configs/osmo/augmentation.yaml | setup -> augmentation | Image-edit augmentation only, no captioning |

| auto_labeling | assets/configs/osmo/auto_labeling.yaml | setup -> auto_labeling | Captioning only on pre-augmented person crops |

Pick the right workflow for the user's request

| User intent | Workflow |

|---|---|

| "Augment person crops and generate captions" / "full PAS pipeline" | e2e |

| "Generate clothing variations" / "augment only" / "image edit" | augmentation |

| "Caption augmented images" / "generate search queries" / "label only" | auto_labeling |

Disambiguation: handle vague requests before committing

Default to autonomy: ask only when missing information blocks execution.

Autonomous defaults (do NOT ask)

• If flow is not explicitly requested, default to e2e.
• If cookbook is not specified, default to default.
• If n_augmentations is not specified, default to 3.
• After any stage completes successfully, continue to the next stage immediately.

Triggers that should pause for disambiguation

| Missing input | Why it matters | Ask |

|---|---|---|

| USER_INPUT_REQUIRED from preflight | Required secret is missing | Ask one concise unblock question |

| Storage backend prefix cannot be derived | Wrong scheme causes runtime storage auth mismatch | "What is the backend-native root prefix for this run?" |

| No ONLINE GPU pool/platform | Workflow cannot schedule | "Which GPU pool/platform should this run target?" |

| NIM deploy fails and no external URLs given | Workers cannot connect to models | "Provide Image Edit / VLM / LLM endpoint URLs, or grant GPU capacity for the NIM operator deploy." |

Step 0: Select Flow and Gather Inputs

Input data policy

• PAS requires person-crop images organized as <person_id>/<view>.jpg subdirectories.
• Always preserve user-provided dataset inputs as first-class.
• Never replace an explicit user dataset with demo assets.
• If no dataset is provided, ask for one (PAS has no built-in demo dataset).

Collect only missing values:

• Dataset source (storage_url + dataset name).
• Flow (e2e, augmentation, auto_labeling); default to e2e.
• OSMO gpu_platform (auto-select when unambiguous).
• Endpoint URLs for Image Edit, VLM, and LLM — optional; default to in-cluster

NIMs and only set for external endpoints.

• Number of augmentations per person ID (default: 3).

Generate run stamp before each submit:

STAMP=$(cat /proc/sys/kernel/random/uuid | cut -c1-8)
RUN_ID="run-$STAMP"

Execution Time Overview (required before run)

Before running any mutating command, provide a short ETA overview.

Baseline ranges:

| Phase | Typical duration |

|---|---|

| Credentials + preflight | ~1-2 min |

| Workflow submit + queue/start | ~1-3 min |

Workflow runtime (depends on dataset size and endpoint latency):

| Flow | Per-image time | Typical dataset (100 images, 3 augs) |

|---|---|---|

| augmentation | ~2.5-3 min/image | ~4-5 hours |

| auto_labeling | ~1-2 min/image | ~2-3 hours |

| e2e | ~3.5-5 min/image | ~6-8 hours |

Common Preconditions (all flows)

• Credential and control-plane preflight

   bash scripts/preflight_credentials.sh --workflow assets/configs/osmo/<flow>.yaml

If output contains USER_INPUT_REQUIRED:, ask one concise unblock question.

• Storage interpolation policy

storage_url must be derived from the actual dataset/upload backend.

Never silently default to stale values on mismatched backends.

• Inference policy (non-negotiable)

• Reuse healthy in-cluster persistent NIM endpoints by default

(qwen-image-edit-2511, qwen3-vl, qwen25-14b).

• If missing/unhealthy, deploy automatically — this is a prerequisite, not a

user decision. Do NOT pause to ask. See references/nim/README.md for the

image-edit NIMService manifest and the VLM/LLM NIM operator install.

• PAS does NOT launch inference servers inside the OSMO workflow; workers

consume the image_edit_url / vlm_url / llm_url endpoints.

• External endpoints are opt-in only (explicit request or explicit URLs);

only then override the *_url values at submit.

• Never scale down/delete existing NIMs to free GPUs.

Submit (all flows)

Every flow uses the same submit shape; only the workflow YAML changes.

SKILLS_DIR="$(cd "$(git rev-parse --show-toplevel)/skills/physical-ai-people-attribute-search" && pwd)"
STAMP=$(cat /proc/sys/kernel/random/uuid | cut -c1-8)
osmo workflow submit assets/configs/osmo/<flow>.yaml \
  --pool <pool> \
  --set-string \
    dataset=<dataset> \
    run_id=run-$STAMP \
    storage_url=<backend-prefix> \
    gpu_platform=<gpu-platform> \
    skills_dir="$SKILLS_DIR"

Endpoints default to the in-cluster NIMs (image_edit_url / vlm_url /

llm_url); deploy/reuse them per the Inference policy above. Do not pass these

unless using external endpoints.

Compatibility note:

• Use exactly one --set-string flag and pass all key/value pairs after it.
• Do not repeat --set/--set-string flags in the same command.

Common optional overrides (append to the same --set-string list):

cookbook=<cookbook_name> \
n_augmentations=<count> \
image_edit_url=<image-edit-endpoint> \
vlm_url=<vlm-endpoint> \
llm_url=<llm-endpoint>

OSMO Monitoring

# Workflow status + task states
osmo workflow query <workflow_id> --format-type json \
  | jq '{status, tasks: [.groups[].tasks[] | {name, status, exit_code}]}'

# Logs for a specific task
osmo workflow logs <workflow_id> --task <task_name> -n 200

# Output retrieval
osmo data list --no-pager <output_url>
osmo data download <output_url> <local_dir>/

For runs expected to exceed two minutes, send heartbeat updates at least every

two minutes.

Post-Run Output

After successful completion, the output directory contains:

For augmentation / e2e:

• <person_id>/aug_<n>/output.jpg — augmented multi-pane image
• <person_id>/aug_<n>/output.txt — natural-language caption
• <person_id>/aug_<n>/output_metadata.json — verification results
• dataset/augmented_data.json — structured dataset with attributes and queries
• dataset/augmented_imgs/ — split per-view crops

For auto_labeling:

• caption_<id>/task/open_qa.json — person-attribute captions grouped by question bank

Supporting files

Use these canonical locations:

• Workflows: assets/configs/osmo/*.yaml
• Runtime scripts: scripts/*.sh
• Flow walkthroughs: references/flows/*.md
• Setup and triage: references/setup.md, references/troubleshooting.md
• Images: references/container-images.md
• Cookbook tuning: assets/cookbooks/default/README.md