# Run DeepSeek Locally in 2026: Ollama, vLLM & GGUF | Dexity

> Run DeepSeek locally: which R1 distill fits your GPU, from the 8B on 8GB cards to the 32B on an RTX 4090. MIT-licensed setup guide.

AI at Work

# How to Run DeepSeek Locally in 2026: R1 Distills, the 671B MoE, GGUF, Ollama, llama.cpp & vLLM

DeepSeek is MIT-licensed and runnable on your own hardware — the question is which one. The realistic local option is the DeepSeek-R1 distill family (1.5B to 70B), which behaves like the Qwen/Llama models they're built on: the 8B fits an 8GB card, the 32B fits a single RTX 4090 (~20GB at Q4\_K\_M). The full 671B-parameter MoE (37B active per token) is a different beast — its Q4\_K\_M GGUF is ~404GB, so it needs a big unified-memory box or a GPU cluster, unless you use Unsloth's dynamic 1.58-bit quant that shrinks it to ~131GB. This guide gives the VRAM-by-model table, what Q4\_K\_M buys you, which tool to use (Ollama, llama.cpp, vLLM, SGLang), the real commands, and when to self-host versus call the API.

17 September 2026

11 min read

Author

[Abhinav Rawat](https://www.linkedin.com/in/abhinav-rawat-8142b4b6/)

Senior Product Manager, Dexity

![A builder slots a GPU into a home PC at night while the monitor shows a terminal running a quantized R1 distill locally, a VRAM meter and an offline badge.](/assets/intel-run-deepseek-locally-hero-feccb6.webp)

On this page

1.  [How do you run DeepSeek locally?](#how-do-you-run-deepseek-locally)
2.  [Which DeepSeek model should you run locally?](#which-deepseek-model-should-you-run-locally)
3.  [The hardware reality: VRAM by model and quant](#the-hardware-reality-vram-by-model-and-quant)
4.  [GGUF and quantization: what Q4\_K\_M actually buys you](#gguf-and-quantization-what-q4km-actually-buys-you)
5.  [Can you run the full 671B DeepSeek?](#can-you-run-the-full-671b-deepseek)
6.  [Which hardware actually runs which DeepSeek?](#which-hardware-actually-runs-which-deepseek)
7.  [Which tool: Ollama, llama.cpp, vLLM, or SGLang?](#which-tool-ollama-llama-cpp-vllm-or-sglang)
8.  [How to run it, step by step](#how-to-run-it-step-by-step)
9.  [Performance: what to expect](#performance-what-to-expect)
10.  [What can go wrong when self-hosting DeepSeek](#what-can-go-wrong-when-self-hosting-deepseek)
11.  [When to self-host vs. use the API](#when-to-self-host-vs-use-the-api)
12.  [Related reading](#related-reading)
13.  [Build the infra skills to run models like this](#build-the-infra-skills-to-run-models-like-this)

Key facts

-   **DeepSeek-V3 and R1 are 671B-parameter MoE models with 37B active per token** (8 of 256 experts routed), Multi-head Latent Attention, and a 128K context window — **MIT-licensed** (weights and code).
-   **The six R1 distill models (1.5B, 7B, 8B, 14B, 32B, 70B)** are dense and are what actually fits on one GPU — built on Qwen and Llama bases.
-   **The 32B distill needs ~20GB of VRAM** (it _is_ Qwen2.5-32B) — a single RTX 4090, matching our Qwen local-run numbers.
-   **The full 671B at Q4\_K\_M is ~404GB**; Unsloth's _dynamic_ 1.58-bit quant shrinks it to **~131GB (~80% smaller)** while staying functional — naive 1-bit quantization breaks it.
-   **A Q4\_K\_M GGUF cuts VRAM ~60% versus FP16** at minimal quality loss — the default download for single-GPU use.
-   **SGLang is DeepSeek's own recommended serving engine**; vLLM also supports V3/R1 natively.

## How do you run DeepSeek locally?

Pick the variant that fits your hardware. The realistic local option for most people is the **DeepSeek-R1 distill family** — dense models from 1.5B to 70B that run like the Qwen and Llama models they're distilled from: the **8B fits an 8GB card**, and the **32B fits a single RTX 4090** (~20GB at a 4-bit quant). The **full 671B-parameter DeepSeek-V3/R1** is a Mixture-of-Experts model with only 37B active per token — powerful, but its 4-bit GGUF is **~404GB**, so it needs a large unified-memory machine or a GPU cluster (or Unsloth's dynamic 1.58-bit build at ~131GB). Everything is **MIT-licensed**, so nothing leaves your machine and there's no per-token bill. Below: exactly which model, what hardware, which tool, and the commands.

## Which DeepSeek model should you run locally?

DeepSeek's lineup causes more confusion than any other open model, so start here. Two families matter:

| Model | Type | Params (total / active) | Local reality |
| --- | --- | --- | --- |
| **R1 distills** (1.5B–70B) | Dense | 1.5B – 70B | The realistic single-GPU option |
| **DeepSeek-V3 / R1 / V3.1 / V3.2** | MoE | 671B / 37B active | Big-RAM box or GPU cluster only |

For almost everyone, the answer is a **distill**: `R1-Distill-Qwen-32B` for max quality on one 24GB card, or `R1-Distill-Llama-8B` / `Qwen-7B` for anything smaller. The **full 671B** (V3, the R1 reasoning model, the V3.1 hybrid, or the V3.2-Exp sparse-attention build) is worth knowing about but is a data-center model, not a desktop one.

Ignore the "DeepSeek V4 / V4-Flash / V4-Pro" specs floating around SEO blogs — there's no primary DeepSeek source (Hugging Face, the API docs, or the papers) for a "V4" at the time of writing, and the numbers those pages quote contradict each other. Anchor on the confirmed V3/V3.1/V3.2 + R1 + distills; verify anything newer against DeepSeek's own repo before trusting it.

## The hardware reality: VRAM by model and quant

The distills are what you'll actually run. Practical targets at a 4-bit (Q4\_K\_M) GGUF — directional estimates, not official figures:

| Distill model | Base | ~VRAM (Q4\_K\_M) | Runs on |
| --- | --- | --- | --- |
| R1-Distill-Qwen-1.5B | Qwen2.5-Math-1.5B | ~1–2 GB | Any 8GB GPU / CPU |
| R1-Distill-Qwen-7B | Qwen2.5-Math-7B | ~5–6 GB | RTX 3060 / 4060 |
| R1-Distill-Llama-8B | Llama-3.1-8B | ~6–8 GB | 8GB card |
| R1-Distill-Qwen-14B | Qwen2.5-14B | ~9–12 GB | RTX 4070 12GB |
| R1-Distill-Qwen-32B | Qwen2.5-32B | ~18–20 GB | RTX 4090 24GB |
| R1-Distill-Llama-70B | Llama-3.3-70B | ~40–43 GB | 2× 3090/4090 · Mac 128GB · A100/H100 |

Two rules of thumb: leave **20–30% of VRAM free for the KV cache**, and on Apple silicon count on **unified memory** (a 128GB Mac runs the 70B distill).

## GGUF and quantization: what Q4\_K\_M actually buys you

For local use you download a **GGUF** — a single-file, quantized build that llama.cpp and Ollama load directly — not the original weights.

-   **Q4\_K\_M** is the default: it cuts VRAM roughly **60% versus FP16** with negligible quality loss. Start here.
-   **Q5\_K\_M / Q6\_K** trade a few GB for a slight quality bump.
-   **Q8\_0** is near-lossless but large.
-   **Q3 and below** save memory but degrade — a fallback, not a default.

The practical move is the same as for any model: **pick the largest model that fits at Q4, not a smaller model at Q8.** A 32B distill at Q4 beats an 8B at Q8 on almost everything.

## Can you run the full 671B DeepSeek?

Yes — but you need to be clever about it. The native FP8 weights are ~671–700GB, and even a Q4\_K\_M GGUF is **~404GB**, which realistically means a ~512GB unified-memory box or a multi-GPU cluster. The trick that makes it runnable off a cluster is **Unsloth's dynamic quantization**, which shrinks the 671B from ~720GB to **~131GB (about 80% smaller)** while keeping it functional:

| Unsloth dynamic quant | ~Size | Note |
| --- | --- | --- |
| IQ1\_S (1.58-bit) | ~131 GB | Smallest that stays coherent |
| IQ2\_XXS (2.22-bit) | ~183 GB | Can edge higher-bit builds on some evals |
| Q2\_K\_XL (2.51-bit) | ~212 GB | More headroom |

The insight is that **calibration matters more than bit-count**: Unsloth keeps the first few layers and attention modules at 4–6 bit and pushes only the bulk MoE layers to ~1.5-bit. A _naive_ all-layer 1-bit quant of DeepSeek produces loops and gibberish — the selective version doesn't. The 131GB build runs in ~160GB of VRAM (2× H100) at ~140 tokens/sec, or on a big-RAM box with the MoE experts offloaded to CPU. (Quant sizes shift between R1 revisions — confirm current numbers on the Unsloth Hugging Face repo.)

Your **RAM + VRAM together must roughly cover the quant's size** (plus KV cache). Fall short and the model spills to disk and crawls to a few tokens per minute. The 671B is a memory problem first, a compute problem second.

## Which hardware actually runs which DeepSeek?

| You have | You can run |
| --- | --- |
| An 8GB GPU | R1 distills up to ~8B |
| RTX 4090 (24GB) | R1-Distill-Qwen-32B |
| 2× 24GB cards, or a 128GB Mac | R1-Distill-Llama-70B |
| ~160GB (2× H100) or a big-RAM box | Full 671B via Unsloth dynamic 1.58-bit |
| 8× H100 / H200 | Native FP8 671B at full 128K context |

## Which tool: Ollama, llama.cpp, vLLM, or SGLang?

| Tool | Use it when | Trade-off |
| --- | --- | --- |
| **Ollama** | You want it running in 30 seconds | Less control; wraps llama.cpp |
| **llama.cpp** | You need granular offload / quant control | More setup |
| **vLLM** | You're serving a team on native weights | Needs full weights + more VRAM |
| **SGLang** | High-throughput serving — **DeepSeek's recommended engine** | Heavier to operate |

For a quantized distill on your own box, **Ollama** is the fastest start. For serving the native FP8 671B on a cluster, **SGLang** (DeepSeek's own recommendation) or vLLM.

## How to run it, step by step

**Option A — Ollama (easiest):**

```
ollama run deepseek-r1:8b     # an R1 distill
ollama run deepseek-r1:32b    # the 32B distill — fits a 4090
```

You get a chat prompt and an OpenAI-compatible endpoint at `http://localhost:11434/v1`.

**Option B — llama.cpp (control):** download a distill GGUF (e.g. a Q4\_K\_M), then:

```
llama-server -m DeepSeek-R1-Distill-Qwen-32B-Q4_K_M.gguf -c 8192 -ngl 99
```

`-ngl 99` offloads all layers to GPU; `-c 8192` sets context — lower it if you hit out-of-memory. For the full 671B on a big-RAM box, keep the MoE experts on CPU (llama.cpp's MoE-offload flags — confirm the current syntax against the Unsloth tutorial).

**Option C — vLLM / SGLang (native, cluster):**

```
vllm serve deepseek-ai/DeepSeek-R1 --tensor-parallel-size 8 --trust-remote-code
```

V3/R1 ship _already in FP8_, so don't re-quantize them; serve across 8 GPUs with tensor parallelism.

## Performance: what to expect

On a single RTX 4090, rough single-user throughput by distill size:

| Setup | Throughput (rough) |
| --- | --- |
| R1-Distill-8B (Q4\_K\_M) | 60–100+ tokens/sec |
| R1-Distill-32B (Q4\_K\_M) | 15–25 tokens/sec |
| Full 671B, Unsloth 1.58-bit, 2× H100 | ~140 tokens/sec |

The honest note: a full 671B on a big-RAM box (experts offloaded to CPU) is **usable for batch and agentic work, not instant chat** — the memory bandwidth caps it. The distills are where you get snappy interactive speed.

## What can go wrong when self-hosting DeepSeek

-   **KV-cache out-of-memory at long context.** 128K context is a huge KV cache — keep context modest, especially on the 671B.
-   **Undersized memory → disk offload.** If RAM+VRAM can't cover the quant, it runs off disk and crawls. Size the box to the model.
-   **Confusing total vs. active params.** The 671B uses 37B per token for _speed_, but you still need memory for the whole 671B. It's fast, not small.
-   **Downloading BF16 by habit.** Grab the GGUF (or serve the native FP8), not the raw BF16 weights.
-   **Re-quantizing FP8 weights.** V3/R1 are already FP8 — don't pass a quantization flag over them.
-   **Unofficial mirrors.** Pull weights from DeepSeek's or Unsloth's official repos, not random re-uploads.

## When to self-host vs. use the API

Self-host when **data can't leave your environment** (regulated industries, private code), when you want **zero per-token cost** at volume, or when you need a **distill running on hardware you already own**. Use the hosted API when you want **full-671B quality without ~130–400GB of memory**, when volume is low enough that per-token pricing beats a cluster, or when you're just building and don't want to operate infrastructure.

## Related reading

-   [How to run Kimi K3 locally](/intel/run-kimi-locally-2026) — the same playbook for a trillion-parameter MoE model.
-   [How to run Qwen 3 locally](/intel/run-qwen-locally-2026) — the dense/MoE family the DeepSeek distills are built on.
-   [The 2026 AI infrastructure stack](/intel/ai-infrastructure-stack-2026) — the production serving stack around models like these.

## Build the infra skills to run models like this

Self-hosting a model like DeepSeek — especially the 671B — is real infrastructure engineering: quantization, multi-GPU serving, KV-cache and memory management, reliability. Dexity's AI Platform Engineering course builds exactly that muscle, hands-on, so standing up a local DeepSeek deployment is a project you run, not just read about.

_Model specs, VRAM figures, and tooling verified against [DeepSeek's V3 report](https://arxiv.org/abs/2412.19437), the [R1](https://huggingface.co/deepseek-ai/DeepSeek-R1) and [V3.2-Exp](https://huggingface.co/deepseek-ai/DeepSeek-V3.2-Exp) model cards, [Unsloth's dynamic-GGUF guide](https://unsloth.ai/blog/deepseekr1-dynamic), the [Ollama library](https://ollama.com/library/deepseek-r1), and SGLang's DeepSeek docs (verified September 2026). Distill VRAM and tokens/sec are directional estimates; open-model versions, quant sizes, and tooling move quickly — confirm current builds before committing hardware. · Dexity.com_

## Frequently asked questions

What hardware do you need to run DeepSeek R1 locally?

For the distills: an 8GB GPU runs up to the 8B, an RTX 4090 (24GB) runs the 32B, and 2× 24GB cards or a 128GB Mac run the 70B. The full 671B needs ~130GB+ (Unsloth dynamic 1.58-bit) up to a multi-GPU cluster for native FP8.

Which DeepSeek model is best for local use?

R1-Distill-Qwen-32B for the best quality on a single 24GB card, or the 7B/8B distills for smaller machines. The full 671B is a data-center model, not a desktop one.

Can you run DeepSeek 671B locally?

Yes, with a big-RAM box or GPU cluster — its Q4\_K\_M GGUF is ~404GB. Unsloth's dynamic 1.58-bit build (~131GB) makes it runnable on ~160GB of VRAM (2× H100) or a large-RAM machine with CPU offload.

Is DeepSeek free to run locally?

Yes. DeepSeek-V3, R1, and the distills are MIT-licensed (the distills also inherit their Qwen/Llama base terms), so you can run, modify, and deploy them locally at no per-token cost.

Ollama or vLLM for DeepSeek?

Ollama for a single user running a quantized distill in seconds. vLLM (or SGLang, DeepSeek's recommended engine) for serving native weights to a team at high throughput.

What's the difference between R1 and the R1 distills?

R1 is the full 671B MoE reasoning model. The distills are smaller _dense_ models (Qwen/Llama bases) trained on R1's outputs — far cheaper to run locally, with much of the reasoning behavior but not the full capability.

## More in AI at Work

[All Intel →](/intel)

[AI at Work

### AI Agent Frameworks in 2026: LangGraph vs CrewAI vs AutoGen vs OpenAI Agents SDK, Compared

There's no single "best" AI agent framework — the right pick depends on your stack and how much control you need. In 2026 the main options are LangGraph (graph-based, most control, strongest for production and complex multi-agent), CrewAI (role-based "crews," fastest to prototype), AutoGen/AG2 (conversation-driven multi-agent), and the OpenAI Agents SDK (lightweight, handoff-based, Python + TypeScript), plus LlamaIndex Workflows, Google ADK, Pydantic AI (type-safe), and the Microsoft Agent Framework (the AutoGen + Semantic Kernel successor). Many teams skip frameworks and build agents with the model SDK plus a loop — Anthropic and Microsoft both recommend starting there. This guide compares them on language, abstraction, multi-agent support, state/memory, and best-fit, with version and maturity claims attributed and dated (they drift).

17 September 2026 · 12 min read](/intel/ai-agent-frameworks-2026) [AI at Work

### How to Build an AI Agent in 2026: A Practical Guide for Engineers

You build an AI agent by wrapping a large language model in a loop: the model gets a goal, decides an action, calls a tool, reads the result, and repeats until the task is done. The core pieces are a controller model, tools defined as JSON schemas, a memory system, an orchestration loop, and a termination condition. Tools are exposed through function-calling APIs — the model returns a structured call, your code runs it, and you feed the output back. The hard part isn't the loop; it's knowing when NOT to build an agent. Most production systems are workflows with predefined paths, not autonomous agents — and Gartner projects over 40% of agentic-AI projects will be canceled by end of 2027 on cost and unclear value. This guide walks the architecture, tool calling, memory, patterns, failure modes, and deployment.

17 September 2026 · 12 min read](/intel/how-to-build-an-ai-agent-2026) [AI at Work

### How to Evaluate AI Agents: Trajectories, Tool Use, and Task Success (2026 Guide)

Evaluating an AI agent means judging both what it produced and how it got there. Unlike a single LLM call, an agent plans, calls tools, and takes many steps — so a correct final answer can hide a broken path. Combine outcome evaluation (task success rate, final-answer correctness) with trajectory evaluation (tool selection, tool-argument accuracy, step efficiency, error recovery). Score deterministic things with code and reserve LLM-as-judge for subjective quality — while guarding against its position, verbosity, and self-preference biases. Track cost, latency, and step count in the same traces as quality. Run offline evals on a fixed dataset before shipping, then monitor online in production and convert every failure into a permanent regression test. This guide covers the taxonomy, the real agent benchmarks (τ-bench, WebArena, GAIA, SWE-bench, BrowseComp), the tooling, and the mistakes.

17 September 2026 · 12 min read](/intel/how-to-evaluate-ai-agents-2026)

Canonical: https://stg.dexity.com/intel/run-deepseek-locally-2026