Cosmos3 Codebase Nav

Cosmos3 Codebase Navigation

When to use this skill

• Use this skill when an agent is navigating the Cosmos3 package
• Use this skill to answer "where is X", "how do I find the config for Y", or any file-location question
• Use this skill when the user opens or edits cosmos3 files and needs orientation

Path convention

All paths below are relative to this file's location (.agents/skills/cosmos3-codebase-nav/). The repo is laid out as:

• cosmos_framework/ — main training package (data, model, trainer, callbacks, checkpoint, utils, …).
• cosmos_framework/configs/base/experiment/ — vfm (generator) experiment SKUs referenced by [train.train_policy].experiment in the recipe TOMLs.
• cosmos_framework/configs/base/reasoner/experiment/ — vlm (reasoner) experiment SKUs.
• cosmos_framework/inference/ — inference subpackage (args, model, inference engine, defaults, Ray serving, common helpers).
• cosmos_framework/scripts/ — top-level entry-point scripts (train, inference, eval, export_model, convert_model_to_dcp, upsample_prompts, caption_from_video, captions_to_sft_jsonl, action_policy_server, …). Invoked as python -m cosmos_framework.scripts.<name>.
• examples/toml/sft_config/<recipe>.toml + examples/launch_sft_<recipe>.sh — paired SFT recipes (training entry-point input). The shell sources examples/_sft_launcher_common.sh, which forwards into cosmos_framework.scripts.train --sft-toml=....
• cosmos_framework/configs/toml_config/ — pydantic schemas (sft_config.py) and helpers that validate the recipe TOML at load time.

Quick Reference

Where parameters and defaults live

| What you're looking for | File |

| --------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------- |

| Sampling params (num_steps, guidance, shift, fps, etc.) | ../../../cosmos_framework/inference/args.py → SamplingArgs, SamplingOverrides |

| Per-modality default values | ../../../cosmos_framework/inference/defaults/<mode>/sample_args.json |

| Setup params (parallelism, checkpoints, model path) | ../../../cosmos_framework/inference/args.py → OmniSetupArgs, OmniSetupOverrides |

| Common args base classes | ../../../cosmos_framework/inference/common/args.py → ArgsBase, OverridesBase |

| Ray serving parallelism presets | ../../../cosmos_framework/inference/ray/configs/latency.yaml, ../../../cosmos_framework/inference/ray/configs/throughput.yaml |

| Feature flags | ../../../cosmos_framework/utils/flags.py |

| Prompt upsampler system prompt | ../../../cosmos_framework/inference/defaults/prompt_upsampler.txt |

| Video captioner system prompt | ../../../cosmos_framework/inference/defaults/video_captioner.txt |

| SFT recipe TOMLs (paired with examples/launch_sft_*.sh) | ../../../examples/toml/sft_config/<recipe>.toml |

| SFT pydantic schema (validates the recipe TOML) | ../../../cosmos_framework/configs/toml_config/sft_config.py |

| Training experiment SKUs (vfm) | ../../../cosmos_framework/configs/base/experiment/ |

| Training experiment SKUs (vlm / reasoner) | ../../../cosmos_framework/configs/base/reasoner/experiment/ |

| Example inputs | ../../../inputs/omni/t2i.json, ../../../inputs/omni/t2v.json, ../../../inputs/omni/i2v.json, … |

Available modality modes for defaults: text2image, text2video, image2video, image2image, video2video, forward_dynamics, inverse_dynamics, wam.

Config defaults resolution chain

When a user runs inference, default parameter values are resolved in this order:

cosmos_framework/inference/defaults/<mode>/sample_args.json     # 1. Per-modality JSON defaults (num_steps, guidance, shift, fps, etc.)
        ↓
_load_modality_defaults() in cosmos_framework/inference/args.py # 2. Loaded and cached at import time
        ↓
SamplingArgs / SamplingOverrides                      # 3. Pydantic models with field-level validation
        ↓
OmniSampleOverrides.build_sample()                    # 4. Merges user overrides → final resolved args
        ↓
_RESOLUTION_SHIFT_DEFAULTS[model_size, resolution]    # 5. Model+resolution shift override (if user didn't set shift)
        ↓
CLI flags (--guidance, --shift, etc.)                 # 6. User overrides from command line

The _RESOLUTION_SHIFT_DEFAULTS table in ../../../cosmos_framework/inference/args.py (on OmniSampleOverrides) overrides the default shift based on model size and resolution, unless the user explicitly specified --shift.

| Mode | Default file | Key defaults |

| ------------- | --------------------------------------------------------------------------- | ---------------------------------------------- |

| text2image | ../../../cosmos_framework/inference/defaults/text2image/sample_args.json | num_frames=1, guidance=6.0, shift=10.0 |

| text2video | ../../../cosmos_framework/inference/defaults/text2video/sample_args.json | num_frames=189, guidance=6.0, shift=10.0 |

| image2video | ../../../cosmos_framework/inference/defaults/image2video/sample_args.json | num_frames=189, guidance=6.0, shift=10.0 |

Action and video2video modes also have defaults under cosmos_framework/inference/defaults/{image2image,video2video,forward_dynamics,inverse_dynamics,policy}/sample_args.json.

Users can also supply a custom defaults file per-request via the defaults_file field in sample arguments (see ../../../docs/inference.md).

Where to make changes

| Task | Edit |

| ------------------------------- | -------------------------------------------------------------------------------------------------------------------------- |

| Change a built-in default value | ../../../cosmos_framework/inference/defaults/<mode>/sample_args.json |

| Add a new CLI parameter | SamplingArgs + SamplingOverrides in ../../../cosmos_framework/inference/args.py, then add to each sample_args.json |

| Change parallelism presets | ../../../cosmos_framework/inference/ray/configs/latency.yaml or throughput.yaml |

| Add a new script | ../../../cosmos_framework/scripts/ — follow inference.py as the pattern |

Key entry points

| Entry point | How to run |

| ---------------------------------- | -------------------------------------------------------------------------------------------- |

| Batch inference | python -m cosmos_framework.scripts.inference |

| Training | python -m cosmos_framework.scripts.train --sft-toml=examples/toml/sft_config/<recipe>.toml |

| Online serving (Ray) | python -m cosmos_framework.inference.ray.serve |

| Submit to Ray server | python -m cosmos_framework.inference.ray.submit |

| Gradio UI | python -m cosmos_framework.inference.ray.gradio |

| Prompt upsampling | python -m cosmos_framework.scripts.upsample_prompts |

| Model export (HF) | python -m cosmos_framework.scripts.export_model |

| DCP conversion | python -m cosmos_framework.scripts.convert_model_to_dcp |

| Diffusers conversion | python -m cosmos_framework.scripts.convert_model_to_diffusers |

| Video captioning | python -m cosmos_framework.scripts.caption_from_video |

| Captions → SFT JSONL | python -m cosmos_framework.scripts.captions_to_sft_jsonl |

| Action policy server (LIBERO HTTP) | python -m cosmos_framework.scripts.action_policy_server_libero |

| Action policy server (RoboLab WS) | python -m cosmos_framework.scripts.action_policy_server_robolab |

Documentation

| Doc | Covers |

| ----------------------------------- | ---------------------------------------------------------- |

| ../../../AGENTS.md | Commands, rules, key file locations (read this first) |

| ../../../README.md | Overview, quickstart, examples |

| ../../../docs/setup.md | Installation, environment, checkpoints |

| ../../../docs/code_structure.md | Repo layout and per-subpackage tour of cosmos_framework/ |

| ../../../docs/inference.md | Sample args, default values, custom defaults |

| ../../../docs/training.md | SFT / post-training workflow |

| ../../../docs/faq.md | FAQ, tips, and troubleshooting |