Hosting a 284B model on 90GB of VRAM

I have an AMD Strix Halo box – a Beelink GTR9 Pro AMD Ryzen™ AI Max+ 395 Processor (I bought it while it was still somewhat affordable) with 128 GB of unified memory. On paper that’s a lot of VRAM, but so far the biggest model I could host that actually had good enough quality to use as a daily driver was Qwen3.6-27B.

I’ve been playing around with larger models a lot recently, but could never get them running.

Then one day recently, while talking to a colleague, it clicked for me. I don’t have to fit the entire model into VRAM if speed isn’t my primary concern; I just have to make it “good enough” so my agentic background tasks can do their thing during the day.

So I set out to host DeepSeek-V4-Flash on my machine and just see how far I can go.

My Hardware Setup

Fail #1: Speculative-decoding Flag

My first launch command had --spec-type draft-mtp in it, because I’d copied the run script from my Qwen3.6-27B model. The container crash-looped immediately:

context type MTP requested but model doesn't contain MTP layers
common_speculative_init_result: failed to create MTP context
llama_server: exiting due to model loading error

DeepSeek-V4-Flash doesn’t ship MTP layers, so requesting speculative decoding fails, of course. To be honest, this was just an oversight on my part, especially since I think MTP would even further tank performance in my case, but I guess that is beside the point.

Fail #2: Docker memory limit

I wanted the model memory-mapped from disk (so the OS could page it in and out of the big unified memory). The container came up … and then died. Over and over.

$ docker events --filter container=<id>
... container oom ...
... container die   exitCode=137 ...
... container start ...        # restart:always brings it back, repeat

Exit 137, so it was the kernel’s OOM killer (SIGKILL). After a bit of debugging, I saw there was a Docker memory limit set to 30 GiB.

First stable config, pure CPU

--n-gpu-layers 0 --mmap

That boots. It runs. It generates ~1.3 tok/s paging weights from the SSD. Slow, but at least it’s alive?

Making it Faster: Offloading Layers to the GPU

Meanwhile, we have 96 GiB of VRAM sitting almost empty. My first idea was just putting the initial dense layers into GPU memory so all of the experts were still paged in and out, but the initial part that is guaranteed to be called for every token is always loaded. In the end, I decided to sweep the parameters instead and just see how many layers I could get away with.

Maybe surprisingly, maybe unsurprisingly, the data is quite monotonic. That’s about 3.2 GiB of VRAM per layer, and generation went from 1.37 to 3.48 tok/s, which is a nice ~2.5x increase.
My initial goal was offsetting the work of loading the initial layers into GPU memory, but -ngl N offloads the last N layers, not the first. I’d assumed it offloaded from layer 0, but the logs said otherwise (layer 0 → CPU, layer 40 → Vulkan0).

Fail #3: –cpu-moe

Digging further into the docs, I found the --cpu-moe option, which keeps the sparse expert weights on the CPU and puts everything that’s used on every token (attention, dense layers, routers) on the GPU. In fact, it loaded just fine using just 15.3 GiB of VRAM.

But, for whatever reason, the first generation request always returned an empty response. Every time. I was very confused by that, so after a bit more debugging, I gave up on that for now.

Fail #4: The „Obvious“ Prefill Optimization That Made Things Worse

Inference speed was fine enough by now for what I need, but when I pointed a real agent harness at it with big prompts, requests timed out every time. The logs showed that prompt processing (prefill) of a 13K-token prompt took ~20 minutes. Prefill activates many experts across all prompt tokens, so I guess I should have expected as much.

The textbook fix is a bigger micro-batch (-ub), which usually speeds up prefill. So I set -ub 2048. The container crash-looped:

failed to allocate Vulkan_Host buffer of size 418478772224 # ~418 GB
graph_reserve: failed to allocate compute buffers

Because Flash Attention is disabled (this seems to be a limitation of my non-CUDA backend at the moment), the attention scratch buffer scales with ubatch × context and gets allocated in that same 30 GiB of system RAM. -ub 2048 at 1M context asks for ~418 GB.

So I ran another parameter sweep and landed on a saner 256K context:

Bigger ubatch is slower here. On a RAM-starved mmap setup, larger compute buffers steal the system RAM that was page-caching your weights, so you thrash *more*.

The default -ub 512 wins.

The actual win was dropping the context. Going from 1M to 256K roughly doubled prefill (to ~24 tok/s). The giant 1M KV + buffers were hogging RAM that’s better spent caching weights. My real prompts are usually under 30K tokens, so 256K is plenty of headroom.

The thing that made it all actually usable

In my use case, luckily, prefill speed barely matters.

An agent reuses almost its entire context every turn (system prompt + history). llama.cpp’s prompt cache serves that common prefix from KV and only prefills the handful of new tokens each turn. The scary 20- or even 60-minute cold prefill happens essentially once; after that, cache hits carry you. In the end, my harness went from timing out to working great with no further changes – the hit rate does the heavy lifting.

The final config

docker run -d \
  --name llama-deepseek-v4-flash \
  --restart always \
  --device /dev/dri/renderD128:/dev/dri/renderD128 \
  --device /dev/dri/card0:/dev/dri/card0 \
  --volume "$(pwd)/.cache":/root/.cache \
  -p 1234:8080 \
  ghcr.io/ggml-org/llama.cpp:server-vulkan \
  -hf unsloth/DeepSeek-V4-Flash-GGUF:UD-Q8_K_XL \
  --n-gpu-layers 24 \
  -np 1 \
  --ctx-size 262144 \
  --mmap

What that gets you on a Strix Halo:

• A 284B / Q8 (162 GB) model running locally
• ~3 tok/s generation, 256K context
• Prefill amortized away by prompt caching for agentic workloads
• ~89 GiB VRAM used, ~7 GiB to spare

Would I recommend it? For interactive chat? Honestly no. It’s more of a “start it and go make coffee” experience for a fresh long prompt (or if you enable thinking). But for a background agent that reuses context well, it’s genuinely fine, and there’s something deeply satisfying about a 284B model answering from a mini-PC on your desk.

Next Steps

I really wanna try colibrì and DwarfStar next, just so I can see how they perform on my machine.