Local models

Local models run entirely in your browser via WebLLM + WebGPU — no server, no API key, no data leaving your device. The trade-off is that your GPU’s available memory sets a hard ceiling on what you can run and how long your prompts can be.

Browser requirements

Browser	WebLLM (local)	BYOK / Bridge
Chrome 113+	✅	✅
Edge 113+	✅	✅
Firefox	❌ (no WebGPU)	✅
Safari	❌ (no WebGPU)	✅

WebGPU must be enabled. On most Chromium builds it is on by default; check chrome://flags/#enable-unsafe-webgpu if a model refuses to load.

GPU memory requirements

Local models use your GPU’s VRAM (or shared system RAM on integrated graphics). Each model loads its weights once and keeps them resident; the KV cache grows with prompt + response length.

Model	Min GPU memory	Recommended
TinyLlama 1.1B	2 GB	4 GB
Llama 3.2 1B	2 GB	4 GB
Phi-3.5 Mini 3.8B	4 GB	6 GB
Llama 3.2 3B	4 GB	6 GB
Llama 3.1 8B	6 GB	8 GB+

Integrated GPU (shared RAM): Integrated GPUs (Intel UHD, AMD Radeon integrated, Apple M-series) have no dedicated VRAM chip — they borrow from your system RAM pool (called UMA, Unified Memory Architecture). The model weights and the KV cache both draw from this shared pool, so available headroom shrinks as prompts get longer.

Note on the System RAM figure in the hardware panel: The browser API (navigator.deviceMemory) is intentionally capped at 8 GB maximum as a privacy measure to prevent fingerprinting, regardless of your actual installed RAM. If your machine has 16, 32, or 64 GB, the panel still shows 8 GB and the capacity estimate is derived from that capped value — the real usable capacity is higher.

Dedicated GPU: VRAM shown above applies directly. A 6 GB card runs 8B models at moderate context lengths.

Prompt length limits

GPU memory constrains not just the model weights but also the KV cache — the memory used to hold the context (your prompt + the response being generated). Very long prompts or long responses shrink the effective limit further.

Rough safe prompt lengths for integrated GPUs:

Model	Safe input + output
TinyLlama 1.1B	~500 tokens total
Llama 3.2 1B	~1,000 tokens
Phi-3.5 Mini	~2,000 tokens
Llama 3.1 8B	~4,000 tokens (dedicated GPU)

For long documents, use the Bridge (Pro) to route to Ollama on a machine with more memory, or switch to a cloud provider for that session.

”Device was lost” error

What causes it:

Prompt + expected response exceeds GPU memory capacity
Another GPU-heavy tab or application consumed available memory before inference started
The selected model is too large for your hardware
Mesa/Vulkan driver instability on Linux — Intel and AMD integrated GPUs running the Mesa open-source driver can trigger VK_ERROR_DEVICE_LOST under sustained WebGPU load even when memory is plentiful. This is a known driver-level issue unrelated to RAM.

How to fix it:

Shorten your prompt — paste only the relevant excerpt rather than an entire document
Switch models — pick a smaller model from the model selector
Close other GPU tabs — video, WebGL games, or another ContextSage tab
Use a cloud model — Groq’s llama-3.3-70b-versatile or any BYOK key handles arbitrarily long prompts without touching your GPU
Use the Bridge — routes to your local Ollama instance which has direct VRAM access and better memory management than WebGPU

Choosing the right model

Scenario	Recommended
Quick Q&A, short docs	TinyLlama 1.1B or Llama 3.2 1B
Contract review (under 4 pages)	Phi-3.5 Mini or Llama 3.2 3B
Long documents, complex reasoning	Bridge → Ollama, or cloud BYOK
Maximum privacy, no internet	Bridge → local Ollama on same machine

First-load download

WebLLM downloads each model once and caches it in your browser’s Cache Storage (not IndexedDB). Model files are typically 500 MB – 5 GB depending on quantization level. After the first load, subsequent sessions start in seconds.

To clear cached models: Chrome DevTools → Application → Cache Storage → delete the webllm-* entries.