# Run Qwen 3 Locally in 2026: GGUF, vLLM & Specs | Dexity

> Run Qwen 3 locally: pick the right Apache 2.0 variant for your GPU, from 8B on ~6GB VRAM to the 235B MoE flagship.

AI at Work

# How to Run Qwen 3 Locally in 2026: GGUF, llama.cpp, vLLM & the Hardware You Actually Need

Qwen 3 is the most-run open-weight model family of 2026 — Apache 2.0, from a 0.6B model that fits on a laptop to the 235B-A22B Mixture-of-Experts flagship. The trick to running it locally is picking the right variant for your GPU: the 8B needs ~5–6 GB of VRAM, the 32B dense model ~20 GB, and the 30B-A3B MoE is the consumer sweet spot — it fits an RTX 4090 and hits ~25 tokens/sec because only ~3B parameters activate per token. This guide gives the VRAM-by-model table, what Q4\_K\_M quantization actually buys you, which tool to use (Ollama, llama.cpp, vLLM, or SGLang), the real commands, and when to self-host versus just call the API.

4 September 2026

11 min read

Author

[Abhinav Rawat](https://www.linkedin.com/in/abhinav-rawat-8142b4b6/)

Senior Product Manager, Dexity

![A developer at a home desk at night runs an open-weight model locally from a terminal, powered by a desktop PC with a glowing 24 GB GPU and a tokens per second readout.](/assets/intel-run-qwen-locally-hero-185258.webp)

On this page

1.  [How do you run Qwen 3 locally?](#how-do-you-run-qwen-3-locally)
2.  [Which Qwen 3 model should you run locally?](#which-qwen-3-model-should-you-run-locally)
3.  [The hardware reality: VRAM by model and quant](#the-hardware-reality-vram-by-model-and-quant)
4.  [GGUF and quantization: what Q4\_K\_M actually buys you](#gguf-and-quantization-what-q4km-actually-buys-you)
5.  [Which tool: Ollama, llama.cpp, vLLM, or SGLang?](#which-tool-ollama-llama-cpp-vllm-or-sglang)
6.  [How to run it, step by step](#how-to-run-it-step-by-step)
7.  [Performance: what to expect](#performance-what-to-expect)
8.  [What can go wrong when self-hosting Qwen 3](#what-can-go-wrong-when-self-hosting-qwen-3)
9.  [When to self-host vs. use the API](#when-to-self-host-vs-use-the-api)
10.  [Related reading](#related-reading)
11.  [Build the infra skills to run models like this](#build-the-infra-skills-to-run-models-like-this)

Key facts

-   **The Qwen 3 family is Apache 2.0-licensed** and spans dense models (0.6B, 1.7B, 4B, 8B, 14B, 32B) plus Mixture-of-Experts models (30B-A3B and the 235B-A22B flagship).
-   **Qwen3-8B runs in ~5–6 GB of VRAM** at a 4-bit quant — an RTX 3060 12 GB or a 16 GB unified-memory Mac handles it.
-   **Qwen3-32B (dense) needs ~20 GB of VRAM**, which puts it on a single RTX 4090 (24 GB) — but with little room left for context.
-   **Qwen3-30B-A3B (MoE) is the efficiency pick**: ~19–24 GB VRAM on an RTX 4090, ~25 tokens/sec, because only ~3B parameters activate per token.
-   **The 235B-A22B flagship** uses 128 experts (8 active per token), 94 layers, and a 32K native context window (up to 131K with YaRN) — it needs multi-GPU or heavy quantization.
-   **A Q4\_K\_M GGUF quant cuts VRAM roughly 60% versus FP16** at minimal quality loss — it's the practical download for single-GPU setups.
-   **Ollama is the fastest way to start** (one command, OpenAI-compatible API); llama.cpp gives granular control; vLLM and SGLang are for serving a team.

## How do you run Qwen 3 locally?

Pick the variant that fits your GPU, download a GGUF quant, and serve it with Ollama, llama.cpp, or vLLM. That's the whole job — the only real decision is model size. **Qwen 3's 8B model runs in ~5–6 GB of VRAM** (an RTX 3060 or a 16 GB Apple silicon Mac), the **32B dense model needs ~20 GB** (RTX 4090 territory), and the **30B-A3B Mixture-of-Experts model is the local sweet spot** — it fits a single 24 GB card and delivers ~25 tokens/sec because only ~3B of its 30B parameters activate per token. The whole Qwen 3 family is Apache 2.0, so nothing leaves your machine and there's no per-token bill. Below: the exact hardware, quant, and tooling choices, with commands.

## Which Qwen 3 model should you run locally?

The family is unusually broad, which is why it's become the default open model — there's a size for every machine. What matters for local use is dense vs. MoE and total vs. active parameters.

| Model | Type | Params (total / active) | Best for |
| --- | --- | --- | --- |
| Qwen3-0.6B / 1.7B / 4B | Dense | 0.6–4B | Laptops, edge, on-device |
| Qwen3-8B | Dense | 8B | The everyday local default |
| Qwen3-14B | Dense | 14B | Better reasoning, still single-GPU |
| Qwen3-32B | Dense | 32B | Max quality on one 24 GB card |
| Qwen3-30B-A3B | MoE | 30B / ~3B active | Best speed-for-quality on consumer GPUs |
| Qwen3-235B-A22B | MoE | 235B / ~22B active | Frontier-class; multi-GPU or heavy quant |

For most people the answer is **Qwen3-8B** (fast, fits anything) or **Qwen3-30B-A3B** (near-32B quality at a fraction of the compute per token). The dense 32B is the highest-quality single-card option but leaves almost no headroom for context. The 235B-A22B is a data-center model — worth knowing about, not something you run on one desktop.

Qwen also ships hosted-scale variants — Qwen3-Coder-480B-A35B for repo-scale agentic coding and a 1T-parameter-class Qwen3-Max — but those are for APIs and GPU clusters, not local machines. This guide is about what you can actually run yourself.

## The hardware reality: VRAM by model and quant

The single number that decides everything is VRAM. These are practical targets at a 4-bit (Q4\_K\_M) GGUF quant, the most common local choice:

| Model | ~VRAM (Q4\_K\_M) | Runs on |
| --- | --- | --- |
| Qwen3-8B | ~5–6 GB | RTX 3060 12 GB · Mac 16 GB |
| Qwen3-14B | ~10–12 GB | RTX 4070 · Mac 24 GB |
| Qwen3-30B-A3B | ~19–24 GB | RTX 4090 · Mac 32 GB+ |
| Qwen3-32B | ~20 GB | RTX 4090 (tight on context) |
| Qwen3-235B-A22B | 140 GB+ | Multi-GPU / large unified memory |

Two rules of thumb: leave **20–30% of your VRAM free for the KV cache** (the model's working memory for context), and on Apple silicon, count on **unified memory** rather than a discrete GPU — a 32 GB+ Mac runs the 30B-A3B comfortably.

## GGUF and quantization: what Q4\_K\_M actually buys you

You almost never download the original BF16 weights for local use — they're twice the size for quality you can't perceive at chat-length prompts. Instead you download a **GGUF** file: a single-file, quantized format that llama.cpp and Ollama load directly.

-   **Q4\_K\_M** is the default: it cuts VRAM roughly **60% versus FP16** with negligible quality loss for most tasks. Start here.
-   **Q5\_K\_M / Q6\_K** trade a few more GB for a slight quality bump — worth it if you have headroom.
-   **Q8\_0** is near-lossless but large; use it only if VRAM is plentiful.
-   **Q3 and below** save memory but degrade noticeably — a fallback, not a default.

The practical move: pick the largest model that fits at Q4\_K\_M rather than a smaller model at Q8. A 30B-A3B at Q4 beats an 8B at Q8 on almost everything.

## Which tool: Ollama, llama.cpp, vLLM, or SGLang?

| Tool | Use it when | Trade-off |
| --- | --- | --- |
| **Ollama** | You want it running in 30 seconds | Less control; wraps llama.cpp |
| **llama.cpp** | You need granular CUDA/layer offload or an unsupported build | More setup |
| **vLLM** | You're serving a team or batch workloads | Needs full weights + more VRAM |
| **SGLang** | High-throughput serving with structured output | Heavier to operate |

For 95% of local use, **Ollama** is the right starting point — it installs fast, pulls models with one command, and exposes an OpenAI-compatible API so your existing code works unchanged. Move to **vLLM** or **SGLang** only when you're past one user and need real throughput.

## How to run it, step by step

**Option A — Ollama (easiest):**

```
# install Ollama, then:
ollama run qwen3:8b
# or the MoE sweet spot:
ollama run qwen3:30b-a3b
```

That's it — you get a chat prompt and an OpenAI-compatible endpoint at `http://localhost:11434/v1`.

**Option B — llama.cpp (control):** download a Qwen3 GGUF (e.g. a Q4\_K\_M build), then:

```
llama-server -m qwen3-30b-a3b-Q4_K_M.gguf -c 8192 -ngl 99
```

`-ngl 99` offloads all layers to GPU; `-c 8192` sets context length — lower it if you hit out-of-memory.

**Option C — vLLM (serving a team):**

```
vllm serve Qwen/Qwen3-8B --max-model-len 32768
```

vLLM loads the full (unquantized) weights by default, so size your GPU accordingly, and gives you high-throughput batched inference behind an OpenAI-compatible API.

## Performance: what to expect

Speed depends on model, quant, and hardware, but rough single-user expectations on a modern consumer GPU (RTX 4090):

| Setup | Throughput (rough) |
| --- | --- |
| Qwen3-8B (Q4\_K\_M) | 60–100+ tokens/sec |
| Qwen3-30B-A3B (Q4\_K\_M) | ~25 tokens/sec |
| Qwen3-32B (Q4\_K\_M) | 15–25 tokens/sec |

The MoE model's trick is visible here: 30B-A3B runs far faster than the dense 32B despite similar quality, because only ~3B parameters fire per token. On Apple silicon, expect lower throughput than a 4090 but enough for interactive use on the 8B–30B range.

## What can go wrong when self-hosting Qwen 3

-   **KV-cache out-of-memory.** The 32B's ~20 GB GGUF leaves only 3–4 GB on a 24 GB card, so a long prompt overflows and crashes. Fix: use the 30B-A3B MoE instead, drop the quant, or shorten context.
-   **Confusing total vs. active params.** A 30B MoE uses ~3B per token for _speed_, but you still need VRAM for the _whole_ 30B. It's fast, not small.
-   **Downloading BF16 by habit.** Grab the GGUF quant, not the originals, unless you're serving with vLLM.
-   **Context creep.** Native context is 32K on the big MoE; pushing to 131K needs YaRN scaling and much more memory. Don't set a huge `-c` you can't feed.

## When to self-host vs. use the API

Self-host when **data can't leave your environment** (regulated industries, private code, PII), when you want **zero per-token cost** at high volume, or when you need **full control** over the model and its uptime. Use the hosted API when you want the **235B-class or Qwen3-Max quality** without owning GPUs, when volume is low enough that per-token pricing is cheaper than hardware, or when you don't want to operate infrastructure. Most teams end up doing both: a local 8B or 30B-A3B for the bulk of requests, and an API for the occasional frontier-grade query.

## Related reading

-   [How to run Kimi K3 locally](/intel/run-kimi-locally-2026) — the same playbook for a trillion-parameter MoE model.
-   [The 2026 AI infrastructure stack](/intel/ai-infrastructure-stack-2026) — what a production serving stack looks like around models like these.
-   [AI infrastructure & platform engineer, by the numbers](/intel/ai-infra-platform-engineer-2026) — who's hiring for exactly this skill, and what it pays.

## Build the infra skills to run models like this

Self-hosting an open model well is real infrastructure engineering — quantization, GPU memory and KV-cache management, multi-GPU serving, and reliability. Dexity's AI Platform Engineering course builds exactly that muscle, hands-on, so standing up a local Qwen 3 deployment is a project you run, not just read about.

_Model specs, VRAM figures, and tooling verified against [the official Qwen 3 blog](https://qwenlm.github.io/blog/qwen3/) and [Spheron's Qwen3 local-GPU guide](https://www.spheron.network/blog/run-qwen3-locally-gpu-requirements-2026/) (verified September 2026). Open-model versions, quant sizes, and tooling move quickly — confirm current builds before committing hardware. · Dexity.com_

## Frequently asked questions

What are the hardware requirements to run Qwen 3 locally?

At a 4-bit (Q4\_K\_M) quant: Qwen3-8B needs ~5–6 GB of VRAM, the 30B-A3B MoE needs ~19–24 GB (a single RTX 4090 or a 32 GB+ Mac), and the dense 32B needs ~20 GB. The 235B-A22B flagship needs multi-GPU or a large unified-memory machine.

Which Qwen 3 model is best for local use?

Qwen3-8B for speed on any machine, or Qwen3-30B-A3B for the best quality-per-compute on a single consumer GPU — it approaches 32B quality while running much faster because only ~3B parameters activate per token.

Is Qwen 3 free to run locally?

Yes. The Qwen 3 dense and MoE open-weight models are Apache 2.0-licensed, so you can run, modify, and deploy them locally at no per-token cost — you only pay for your own hardware and electricity.

What's the difference between the dense and MoE Qwen 3 models?

Dense models (e.g. 32B) use every parameter on every token. MoE models (30B-A3B, 235B-A22B) route each token through a small subset of "expert" parameters, so they run much faster for their size — but you still need VRAM for the full parameter count.

Ollama or vLLM for Qwen 3?

Ollama for a single user who wants it working in seconds. vLLM (or SGLang) when you're serving a team or batch workloads and need high-throughput, batched inference.

## More in AI at Work

[All Intel →](/intel)

[AI at Work

### AI Agent Frameworks in 2026: LangGraph vs CrewAI vs AutoGen vs OpenAI Agents SDK, Compared

There's no single "best" AI agent framework — the right pick depends on your stack and how much control you need. In 2026 the main options are LangGraph (graph-based, most control, strongest for production and complex multi-agent), CrewAI (role-based "crews," fastest to prototype), AutoGen/AG2 (conversation-driven multi-agent), and the OpenAI Agents SDK (lightweight, handoff-based, Python + TypeScript), plus LlamaIndex Workflows, Google ADK, Pydantic AI (type-safe), and the Microsoft Agent Framework (the AutoGen + Semantic Kernel successor). Many teams skip frameworks and build agents with the model SDK plus a loop — Anthropic and Microsoft both recommend starting there. This guide compares them on language, abstraction, multi-agent support, state/memory, and best-fit, with version and maturity claims attributed and dated (they drift).

17 September 2026 · 12 min read](/intel/ai-agent-frameworks-2026) [AI at Work

### How to Build an AI Agent in 2026: A Practical Guide for Engineers

You build an AI agent by wrapping a large language model in a loop: the model gets a goal, decides an action, calls a tool, reads the result, and repeats until the task is done. The core pieces are a controller model, tools defined as JSON schemas, a memory system, an orchestration loop, and a termination condition. Tools are exposed through function-calling APIs — the model returns a structured call, your code runs it, and you feed the output back. The hard part isn't the loop; it's knowing when NOT to build an agent. Most production systems are workflows with predefined paths, not autonomous agents — and Gartner projects over 40% of agentic-AI projects will be canceled by end of 2027 on cost and unclear value. This guide walks the architecture, tool calling, memory, patterns, failure modes, and deployment.

17 September 2026 · 12 min read](/intel/how-to-build-an-ai-agent-2026) [AI at Work

### How to Evaluate AI Agents: Trajectories, Tool Use, and Task Success (2026 Guide)

Evaluating an AI agent means judging both what it produced and how it got there. Unlike a single LLM call, an agent plans, calls tools, and takes many steps — so a correct final answer can hide a broken path. Combine outcome evaluation (task success rate, final-answer correctness) with trajectory evaluation (tool selection, tool-argument accuracy, step efficiency, error recovery). Score deterministic things with code and reserve LLM-as-judge for subjective quality — while guarding against its position, verbosity, and self-preference biases. Track cost, latency, and step count in the same traces as quality. Run offline evals on a fixed dataset before shipping, then monitor online in production and convert every failure into a permanent regression test. This guide covers the taxonomy, the real agent benchmarks (τ-bench, WebArena, GAIA, SWE-bench, BrowseComp), the tooling, and the mistakes.

17 September 2026 · 12 min read](/intel/how-to-evaluate-ai-agents-2026)

Canonical: https://stg.dexity.com/intel/run-qwen-locally-2026