Huggingface Local Models

Hugging Face Local Models

Search the Hugging Face Hub for llama.cpp-compatible GGUF repos, choose the right quant, and launch the model with llama-cli or llama-server.

Default Workflow

• Search the Hub with apps=llama.cpp.
• Open https://huggingface.co/<repo>?local-app=llama.cpp.
• Prefer the exact HF local-app snippet and quant recommendation when it is visible.
• Confirm exact .gguf filenames with https://huggingface.co/api/models/<repo>/tree/main?recursive=true.
• Launch with llama-cli -hf <repo>:<QUANT> or llama-server -hf <repo>:<QUANT>.
• Fall back to --hf-repo plus --hf-file when the repo uses custom file naming.
• Convert from Transformers weights only if the repo does not already expose GGUF files.

Quick Start

Install llama.cpp

brew install llama.cpp
winget install llama.cpp

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
make

Authenticate for gated repos

hf auth login

Search the Hub

https://huggingface.co/models?apps=llama.cpp&sort=trending
https://huggingface.co/models?search=Qwen3.6&apps=llama.cpp&sort=trending
https://huggingface.co/models?search=<term>&apps=llama.cpp&num_parameters=min:0,max:24B&sort=trending

Run directly from the Hub

llama-cli -hf unsloth/Qwen3.6-35B-A3B-GGUF:UD-Q4_K_M
llama-server -hf unsloth/Qwen3.6-35B-A3B-GGUF:UD-Q4_K_M

Run an exact GGUF file

llama-server \
    --hf-repo unsloth/Qwen3.6-35B-A3B-GGUF \
    --hf-file Qwen3.6-35B-A3B-UD-Q4_K_M.gguf \
    -c 4096

Convert only when no GGUF is available

hf download <repo-without-gguf> --local-dir ./model-src
python convert_hf_to_gguf.py ./model-src \
    --outfile model-f16.gguf \
    --outtype f16
llama-quantize model-f16.gguf model-q4_k_m.gguf Q4_K_M

Smoke test a local server

llama-server -hf unsloth/Qwen3.6-35B-A3B-GGUF:UD-Q4_K_M

curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer no-key" \
  -d '{
    "messages": [
      {"role": "user", "content": "Write a limerick about exception handling"}
    ]
  }'

Quant Choice

• Prefer the exact quant that HF marks as compatible on the ?local-app=llama.cpp page.
• Keep repo-native labels such as UD-Q4_K_M instead of normalizing them.
• Default to Q4_K_M unless the repo page or hardware profile suggests otherwise.
• Prefer Q5_K_M or Q6_K for code or technical workloads when memory allows.
• Consider Q3_K_M, Q4_K_S, or repo-specific IQ / UD-* variants for tighter RAM or VRAM budgets.
• Treat mmproj-*.gguf files as projector weights, not the main checkpoint.

Load References

• Read hub-discovery.md for URL-first workflows, model search, tree API extraction, and command reconstruction.
• Read quantization.md for format tables, model scaling, quality tradeoffs, and imatrix.
• Read hardware.md for Metal, CUDA, ROCm, or CPU build and acceleration details.

Resources

• llama.cpp: https://github.com/ggml-org/llama.cpp
• Hugging Face GGUF + llama.cpp docs: https://huggingface.co/docs/hub/gguf-llamacpp
• Hugging Face Local Apps docs: https://huggingface.co/docs/hub/main/local-apps
• Hugging Face Local Agents docs: https://huggingface.co/docs/hub/agents-local
• GGUF converter Space: https://huggingface.co/spaces/ggml-org/gguf-my-repo