Ollama Agent Skill

Ollama — run open-weight LLMs on one box

Ollama serves GGUF models from a local daemon at http://localhost:11434, exposing both a native

HTTP API and an OpenAI-compatible layer. Your job: reach for the right command, the right endpoint,

and the right quant for the hardware in front of you — and recognize when the model does not fit

and the work belongs on a remote GPU instead.

This skill owns: install/serve, pull/tag, the local API (native + OpenAI-compat), Modelfiles,

quantization choice, and VRAM/RAM sizing on a single machine.

When to use / when not

Use when the model runs on *this* machine: pulling/running a model, fixing an OOM, choosing

Q4 vs Q8, authoring a Modelfile, or wiring an app to localhost:11434.

Go elsewhere when:

• Hosting behind a managed/remote GPU, autoscaling, or serverless inference → runpod, modal,

replicate, together-fireworks, fal. Ollama is local, single-box, no autoscale.

• Downloading raw weights, datasets, hf/transformers, repo management → huggingface.
• Designing chunking / retrieval / reranking around a model → rag or embeddings-search.
• Orchestrating multi-step calls, routing, pipeline evals → llm-pipeline / agent-eval.
• Writing the prompt/system-message *content* itself → prompt-engineering.

(Those siblings live in the catalog by id; link them only once their SKILL.md exists on disk.)

Quickstart

ollama serve                 # start the daemon (a desktop install already runs it)
ollama pull qwen3:8b         # download a model + tag; :8b is explicit — avoid bare :latest
ollama run qwen3:8b          # interactive REPL, or: ollama run qwen3:8b "summarize this"
ollama ps                    # what is LOADED in VRAM right now + when it unloads (keep_alive)
ollama list                  # what is on disk (pulled), not what is loaded
ollama show qwen3:8b         # template, params, context length, quant of a model
ollama rm qwen3:8b           # free disk; ollama stop qwen3:8b unloads from memory

ps vs list is the OOM-debug split: list is disk, ps is memory. A model only eats VRAM once a

request loads it; it unloads after keep_alive (default 5m).

Pick a model + quant

Quantization trades VRAM for quality. The everyday default is Q4_K_M: roughly half the memory of

fp16 for ~3–5% quality loss. Q8_0 is near-lossless at ~1 byte/param. fp16 is the unquantized

ceiling at 2 bytes/param.

Sizing formula (weights only) — a rule of thumb, not a per-model spec sheet:

weights_GB ≈ params(B) × bytes_per_param × 1.2   # ×1.2 = runtime overhead
bytes_per_param:  Q4_K_M ≈ 0.5   Q8_0 ≈ 1.0   fp16 = 2.0
# then ADD the KV cache (see below) — it is NOT in this number.

These bytes/param are conservative round-downs of the measured k-quant rates: llama.cpp's quantize

benchmark reports Q4_K_M ≈ 4.89 bits/weight (~0.6 byte/param) and Q8_0 ≈ 8.5 bits/weight (~1.06

byte/param) on Llama-3.1-8B (llama.cpp quantize README,

accessed 2026-06-02). Rounding to 0.5 / 1.0 keeps the estimate on the safe side; the per-row GB figures

in the table below are derived from this formula, not vendor-published numbers — verify with ollama show.

| VRAM / unified mem | Comfortable choice (Q4_K_M) | Notes |

| --- | --- | --- |

| 8 GB | 7–8B Q4_K_M (~5–6 GB) | leave headroom for KV cache + the OS |

| 12 GB | up to ~14B Q4_K_M (~9–10 GB) | 7–8B at Q8_0 also fits |

| 16 GB | 14B Q4_K_M comfortably; 32B is tight | 32B Q4_K_M ≈ 20 GB — won't fit |

| 24 GB | 32B Q4_K_M (~20 GB) | 70B does not fit at any usable quant |

| 48 GB+ / 2×24 GB | 70B Q4_K_M (~40–48 GB) | needs the full budget; long context pushes over |

| Mac unified (e.g. 64 GB) | weights share RAM with everything else | budget against total unified memory |

KV cache is the trap. It grows ~linearly with num_ctx and lives in VRAM *on top of* the weights.

At long context (e.g. 128K) a 70B can add tens of GB of cache — often more than people budget for. If

you are tight: cap num_ctx, or shrink the cache with OLLAMA_KV_CACHE_TYPE=q8_0 (or q4_0). See

references/hardware-sizing.md for the KV math and a per-context table.

Ollama runs a llama.cpp-backed engine (GGUF) by default, with a scheduler that reduces OOM crashes and

improves multi-GPU placement. On Apple Silicon it can use an MLX backend (shipped in Ollama 0.19,

per ollama.com/blog/mlx, 2026-03-30), but **only on Macs with >32 GB of

unified memory** — below that gate it stays on the llama.cpp engine. None of this invents memory you

don't have: when the box can't hold the model, that's a runpod/modal job, not a quant downgrade.

The API

Two surfaces, same daemon. Use native /api/chat when you want Ollama-specific fields

(keep_alive, format as a JSON schema, think); use the OpenAI-compat /v1 layer to reuse an

existing OpenAI SDK unchanged.

Native chat (/api/chat), non-streaming:

curl http://localhost:11434/api/chat -d '{
  "model": "qwen3:8b",
  "messages": [{"role": "user", "content": "Name three primes."}],
  "stream": false,
  "options": {"temperature": 0.2, "num_ctx": 8192},
  "keep_alive": "10m"
}'

stream defaults to true (NDJSON, one object per line, final object has done: true + timing

stats). options.num_ctx sets the context window *for this request* — it does not persist; bake it

into a Modelfile if you want it permanent.

OpenAI-compatible — point any OpenAI SDK at localhost:11434/v1 with a dummy key:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")  # key is ignored
resp = client.chat.completions.create(
    model="qwen3:8b",
    messages=[{"role": "user", "content": "Name three primes."}],
    temperature=0.2,
)
print(resp.choices[0].message.content)

Structured output — pass a JSON schema as format (native) so the model is constrained to valid JSON:

curl http://localhost:11434/api/chat -d '{
  "model": "qwen3:8b",
  "messages": [{"role": "user", "content": "Extract name and age from: Ana is 30."}],
  "stream": false,
  "format": {
    "type": "object",
    "properties": {"name": {"type": "string"}, "age": {"type": "integer"}},
    "required": ["name", "age"]
  }
}'

Tool calling (tools), multimodal (images as base64), embeddings (/api/embed), and the full field

tables live in references/api.md. Endpoint map at a glance: /api/generate,

/api/chat, /api/embed, /api/create, /api/pull, /api/show, /api/ps, /api/tags.

Modelfiles

A Modelfile bakes a base model + system prompt + parameters into a new named model. Build with

ollama create.

FROM qwen3:8b
SYSTEM "You are a terse senior code reviewer. Answer in bullet points."
PARAMETER num_ctx 16384
PARAMETER temperature 0.2
PARAMETER stop "<|im_end|>"

ollama create reviewer -f Modelfile     # now: ollama run reviewer

• FROM is required — a model tag or a local file (FROM ./model.gguf to import a raw GGUF).
• PARAMETER num_ctx makes the context window permanent (vs the per-request options.num_ctx).
• SYSTEM, TEMPLATE, LICENSE, ADAPTER (LoRA) round out the instruction set.

Quantize on create from an fp16/fp32 source:

ollama create reviewer --quantize q4_K_M -f Modelfile   # FROM must be an fp16/fp32 model

--quantize only works when the FROM source is full-precision; you cannot re-quantize an

already-Q4 model. To go from Hugging Face weights to a GGUF in the first place, that conversion is a

huggingface job — Ollama imports the result.

When to leave the box

If the comfortable-choice row for your VRAM can't hold the model you actually need (e.g. you need 70B

quality on a 12 GB laptop), stop downgrading quant — quality collapses below Q4 and you'll still OOM at

real context. Move it to a remote GPU: runpod (rent a GPU), modal (serverless container + GPU

autoscale), or a hosted endpoint (replicate, together-fireworks, fal). Ollama is the right tool

*until the weights + KV cache exceed the single box*.

Anti-patterns

| Bad | Good | Why |

| --- | --- | --- |

| Pull fp16 on a box that only fits Q4 | Pull Q4_K_M (or Q8_0 if it fits) | fp16 is 4× the VRAM of Q4 for ~3–5% quality; you'll OOM for nothing |

| num_ctx: 128000 on a 12 GB GPU | Cap num_ctx to what fits; OLLAMA_KV_CACHE_TYPE=q8_0 | KV cache scales with context and sits on top of weights — long context dwarfs the model |

| /api/generate for a chat with history | /api/chat with a messages array | generate is single-turn; you'd hand-concatenate history and break the chat template |

| ollama pull mistral:latest, assume it's small | Pin an explicit tag (:7b, a quant tag) and ollama show it | :latest size/quant drifts release to release; sizing breaks silently |

| Treat Ollama as a multi-tenant prod server | Use it local/single-box; scale → runpod/modal | one daemon, limited parallelism (OLLAMA_NUM_PARALLEL); not built for fleet serving |

| Hardcode api.openai.com when target is local | base_url="http://localhost:11434/v1", dummy key | the OpenAI SDK works unchanged against the compat layer; no remote calls, no key leak |

| Downgrade to Q2 to force a 70B onto 12 GB | Pick a model that fits, or move to a remote GPU | sub-Q4 quality drops sharply *and* it still won't fit at real context |

| Assume ollama list means it's loaded | ollama ps for memory, list for disk | a pulled model uses 0 VRAM until a request loads it |

Verify

Run scripts/verify.sh [TARGET] from your project root (or a dir holding a Modelfile). Static by

default — it needs neither Ollama installed nor a running daemon. It lints a Modelfile (FAIL if no

FROM; WARN on unknown instructions or a num_ctx so high it will OOM consumer GPUs), notes whether

app code points at the local localhost:11434 / /v1 endpoint vs only-remote hosts, and — only if

ollama is on PATH — best-effort confirms a model is present (WARN, not FAIL). It exits non-zero

only on a real FAIL; an empty/clean target passes.

References

• references/api.md — full endpoint catalog, request/response field tables,

OpenAI-compat path mapping, structured output, tool calling, streaming, embeddings (curl + Python).

• references/hardware-sizing.md — the full quant ladder, VRAM formula

derivation, KV-cache math + per-context table, per-model chart, Apple Silicon unified-memory notes,

and the env knobs (OLLAMA_KV_CACHE_TYPE, OLLAMA_FLASH_ATTENTION, OLLAMA_NUM_PARALLEL,

OLLAMA_MAX_LOADED_MODELS) for fitting tight boxes.