Dspy Adapters Multimodal Agent Skill
Use for DSPy adapter selection, JSONAdapter, XMLAdapter, ChatAdapter, native function calling, structured outputs, and multimodal inputs like dspy.Image or dspy.Audio.
Install
one command, takes just this skill from the repositorynpx skills add https://github.com/OmidZamani/dspy-skills --skill dspy-adapters-multimodalWhat comes with it
294 bytes besides the instructionThe instruction itself
9 sections, as written by the authorDSPy Adapters and Multimodal I/O
Goal
Choose an adapter deliberately and model image, audio, and file inputs with DSPy's typed primitives.
Adapter Selection
| Adapter | Use it for |
|---------|------------|
| dspy.ChatAdapter() | Default, human-readable field markers, broad model compatibility |
| dspy.JSONAdapter() | Structured JSON output and native function calling where supported |
| dspy.XMLAdapter() | XML-tagged fields when XML is easier for the target LM to follow |
| dspy.TwoStepAdapter() | A separate extraction pass when parsing needs extra help |
Configure globally or for a limited scope:
import dspy
dspy.configure(
lm=dspy.LM("openai/gpt-4o-mini"),
adapter=dspy.JSONAdapter(),
)
with dspy.context(adapter=dspy.XMLAdapter()):
result = dspy.Predict("question -> answer")(question="What is DSPy?")
Native Function Calling
JSONAdapter enables native function calling by default. ChatAdapter keeps text parsing by default. Override either behavior explicitly:
chat_native = dspy.ChatAdapter(use_native_function_calling=True)
json_manual = dspy.JSONAdapter(use_native_function_calling=False)
DSPy falls back to manual parsing when the configured LM does not support native function calling.
Image Inputs
class DescribeImage(dspy.Signature):
image: dspy.Image = dspy.InputField()
description: str = dspy.OutputField()
describe = dspy.Predict(DescribeImage)
result = describe(image=dspy.Image("./diagram.png"))
Pass a local path, HTTP URL, bytes, PIL image, or existing data URI directly to dspy.Image(...).
Audio and File Inputs
class SummarizeAudio(dspy.Signature):
audio: dspy.Audio = dspy.InputField()
summary: str = dspy.OutputField()
audio = dspy.Audio.from_file("./meeting.wav")
summary = dspy.Predict(SummarizeAudio)(audio=audio)
class SummarizeFile(dspy.Signature):
file: dspy.File = dspy.InputField()
summary: str = dspy.OutputField()
document = dspy.File.from_path("./research.pdf")
summary = dspy.Predict(SummarizeFile)(file=document)
Provider capabilities vary. Verify that the selected model accepts the media type before deployment.
Best Practices
- Start with
ChatAdapter; switch only for a measured reason. - Use typed signatures for structured output.
- Test adapter behavior against the exact production model.
- Avoid deprecated
Image.from_file()andImage.from_url()helpers; calldspy.Image(...). - Keep local file handling and uploaded file IDs within provider policy.
Related Skills
- Design signatures: dspy-signature-designer
- Build tool agents: dspy-react-agent-builder
Official Documentation
- Adapters guide: https://dspy.ai/learn/programming/adapters/
- Tools guide: https://dspy.ai/learn/programming/tools/
- XMLAdapter API: https://dspy.ai/api/adapters/XMLAdapter/
- Image API: https://dspy.ai/api/primitives/Image/
- Audio API: https://dspy.ai/api/primitives/Audio/
Other skills for the same job
different authors, same section of the catalogue
Create beautiful visual art in .png and .pdf documents using design philosophy. You should use this skill when the user asks to create a poster, piece of art, design, or other static piece. Create original visual designs, never copying existing artists' work to avoid copyright violations.
Creating algorithmic art using p5.js with seeded randomness and interactive parameter exploration. Use this when users request creating art using code, generative art, algorithmic art, flow fields, or particle systems. Create original algorithmic art rather than copying existing artists' work to avoid copyright violations.
Improves the quality of images, especially screenshots, by enhancing resolution, sharpness, and clarity. Perfect for preparing images for presentations, documentation, or social media posts.
Downloads videos from YouTube and other platforms for offline viewing, editing, or archival. Handles various formats and quality options.
Lightweight WSI tile extraction and preprocessing. Use for basic slide processing tissue detection, tile extraction, stain normalization for H&E images. Best for simple pipelines, dataset preparation, quick tile-based analysis. For advanced spatial proteomics, multiplexed imaging, or deep learning pipelines use pathml.
Microscopy data management platform. Access images via Python, retrieve datasets, analyze pixels, manage ROIs/annotations, batch processing, for high-content screening and microscopy workflows.
Python library for working with DICOM (Digital Imaging and Communications in Medicine) files. Use this skill when reading, writing, or modifying medical imaging data in DICOM format, extracting pixel data from medical images (CT, MRI, X-ray, ultrasound), anonymizing DICOM files, working with DICOM metadata and tags, converting DICOM images to other formats, handling compressed DICOM data, or processing medical imaging datasets. Applies to tasks involving medical image analysis, PACS systems, radiology workflows, and healthcare imaging applications.
This skill should be used when working with pre-trained transformer models for natural language processing, computer vision, audio, or multimodal tasks. Use for text generation, classification, question answering, translation, summarization, image classification, object detection, speech recognition, and fine-tuning models on custom datasets.
How to use it
Take omidzamani/dspy-adapters-multimodal from the repository into ~/.claude/skills for personal
use, or into .claude/skills inside a project.
The agent identifies a skill by the name field in its header. Two skills with the
same name cannot sit side by side — one of them will be ignored.