Round two: Hosting GLM 5.2 via expert streaming 2/2 Alle Serienteile anzeigen

Alle Serienteile

Serie: local-llm-mini-series

Round two: Hosting GLM 5.2 via expert streaming

Teil 2 von 2

GLM 5.2 is a 744-billion-parameter MoE model – way too big for any GPU I own. Expert streaming let me run it anyway, at around 1.2 tokens per second, entirely from a single mini PC.

Blog » AI » Round two: Hosting GLM 5.2 via expert streaming

I finally had time to try colibrì. For the uninitiated, colibrì is an open source project that allows you to host LLMs that you have no business hosting under normal circumstances. It works by allowing you to stream the experts of the supported MoE models ad-hoc when they are needed. Since speed already isn’t as important as quality is for my “offline” tasks, I was wondering how usable GLM 5.2 (the largest available model at time of writing) was going to be.

GLM 5.2 is a 744-billion-parameter Mixture-of-Experts open weights model with about 40 billion active parameters per token.

Giving the RAM back to the CPU

This is the inversion from last time. For the VRAM-resident V4-Flash setup, the 96 GiB carve-out was the largest performance lever. For a CPU-streaming engine like colibrì, though, that would just be 96 GiB the expert cache can’t touch.

So into the BIOS, UMA frame-buffer from 96 GiB down to 512 MB, which seemed like a sensible minimum, which drove available system RAM from ~30 GiB to 124 GiB.

The thing I was actually nervous about was breaking the V4-Flash Vulkan container, since that setup worked and I didn’t want to trade one for the other. Vulkan/RADV pulls GPU memory from GTT, which already had the entire 124 GiB available. As far as I can tell on unified memory, it’s the same physical bytes either way, so I could always fall back to the mmap-ed V4-Flash.

Is SSD streaming even fast enough?

Everything colibrì does is bound by how fast I can pull cold experts off disk. So before tuning anything, I benchmarked the drive the way the engine actually hits it: random reads at roughly expert-block size (~19 MB), with O_DIRECT so I was measuring the SSD and not the page cache.

It’s a Crucial P310, Gen4, DRAM-less. fio said:

patternresult
random 16 MiB read, QD1~6.0 GB/s (2.55 ms/expert)
sequential read6.1 GB/s
sustained write, 100 GB~950 MB/s, dropping to 240 past the SLC cliff

~6 GB/s at expert block size even at queue depth 1. So the disk wasn’t going to be my limit, which was good news, because I can’t make the disk faster. However, I can hand the cache more RAM.

Getting it running

Colibrì either downloads the 756 GB FP8 source and converts it, or pulls a pre-converted int4 container off Hugging Face. I took the pre-converted one: 429 GB, no conversion step, and I wasn’t stuck downloading the model all night.

The engine is a single C file, so setting it up was as easy as:

gcc -O3 -march=native -fopenmp -pthread colibri.c -o colibri -lm

Built in seconds, self-test passed. After the Docker saga last time, this was a nice change.

Fail #1: The engine ignored 90 GB of my RAM

Baseline run, still on the old 30 GiB config: 0.42 tok/s, 12% expert hit rate, ~90% of wall-clock waiting on disk. Exactly the disk-starved picture you’d expect.

Initially, even after I rebalanced to 124 GiB, I was surprised to only see memory utilization of ~30 GiB with 0.42 tok/s, 12% expert hit rate, ~90% of wall-clock waiting on disk. After some digging, I found out that coli run picks a conservative auto budget on its own. It had quietly left 85 GB on the table.

./coli run --ram 110 ...

Used memory climbed to 100 GB, hit rate went from 12% to 61%, and inference speed from 0.42 tok/s to 0.60 tok/s.

Messing with the configuration flags

configdecode
MTP=0 PIPE=1 (the default bahavior)0.65 tok/s
DIRECT=11.08 tok/s
URING=11.13 tok/s
CACHE_ROUTE=1, topp 0.701.79 tok/s
topp 0.502.23 tok/s

The documentation suggests adding DIRECT=1, which bypasses the page cache for expert reads, which on its own was a 66% performance improvement. The docs warn it can be neutral or worse on a DRAM-less drive, which mine is. So I half expected it to do nothing, but I won’t complain that it did.

To be honest, the other parameters I used are just somewhat educated guesses on what might help from reading the performance tuning documentation.

Fail #2: PILOT, the prefetcher for a different machine

PILOT=1 is router-lookahead prefetch, which predicts the next layer’s experts and reads them ahead of compute. On a disk-bound box that sounded to me like it would help; however, it was net-negative – roughly 12% less performance. Evaluating a longer prompt yielded: “1214 prefetches used, 12636 discarded,” which suggests to me that the intelligence just isn’t there yet in the prefetch.

Fail #3: My aggressive config was great in tests and terrible in real life

I convinced myself topp 0.50 was safe by testing it: a coding task (an LCS function I actually executed against test cases, 6/6 passing) and multi-step arithmetic tests. Good enough, so I set it as the default and served it.

The first real chat was literally “Hello :)”. It came back as an infinite loop of the model thinking to itself without ever answering, leaking template tokens, and at one point claiming to be Gemini.

In hindsight, I think that behavior makes sense to me. Code and arithmetic are likely strong suits of the model, so they’re exactly the ones that survive having half the routing mass pruned away. Open-ended chat isn’t. I switched the server to the lossless profile (topp 0.85, no cache-routing), which costs a lot at ~1.2 tok/s but is coherent, and correctly knows it’s GLM 5.2. For something I actually leave running untended, quality-first is the only default that makes sense to me.

Fail #4: The bug that wasn’t mine

Even at lossless, some replies emit a literal <|user|> and hallucinate a whole fake next turn. I assumed the WebUI was mangling something, but I later found out it wasn’t. The raw API had the same issue.

It turned out to be a known issue (#401): in serve mode the engine cuts the stop tokens down to just <|endoftext|> and leaves the role markers to the Python gateway, and on main that gateway filter was incomplete. However, the fix was already on the dev branch, so I just switched branches, rebuilt the C file and the web UI; all prompts stop cleanly now.

I guess this goes to show how very much in development colibrì still is.

Serving it on my network

Two things here cost me longer than they should have. Binding 0.0.0.0 gets every LAN client a 403. There’s an anti-DNS-rebinding host guard that only accepts requests whose host matches the bind address, so you want the actual LAN IP. And it won’t bind off-localhost at all without an API key, either. Fair enough, but could be documented better.

Fail #5: Chasing RAM the engine won’t take

Since my use-case is just me as the sole user, running a single agent, I dropped to 1 KV slot and started handing every spare gigabyte to the expert cache. First 116 GiB, then 120 and 122, and finally, because I was starting to get suspicious – 190.

--rampeak utilization
116120.9 GiB
120120.9 GiB
122120.9 GiB
190120.9 GiB

The peak never moved. Every value “worked” because they were all the same plan underneath. It looks like the engine clamps its cache plan to somewhere around the available memory at process start minus 2 GB and ignores anything past that.

What did help, however, was dropping context from 32K to 16K, which is enough for my workloads. That freed KV reservation, giving me roughly 4 GiB more cached experts.

The final config

MTP=0 URING=1 DIRECT=1 PIPE_WORKERS=16 \
  ./coli serve --model /nvme/glm52_i4 \
  --ram 122 --topp 0.85 --ctx 16384 \
  --host 192.168.x.y --port 8000 --api-key "$KEY" \
  --kv-slots 1

Which gets me:

  • GLM-5.2, 744B at int4 (429 GB), ~100 GB of experts resident, the rest are streamed
  • ~1.2 tok/s after a short warmup period
  • ~71% expert hit rate during coding tasks
  • ~70% of wall-clock is disk wait

Would I recommend it? Honestly, the same result as last time; for interactive chat, 1.2 tok/s is just too slow. But for offline/scheduled workloads it’s perfectly usable.

What I am looking forward to

The colibrì maintainers are insane, adding features (or I guess letting features be added by their agents?) almost every single day.

Currently they are adding support for DeepSeek-V4-Flash. With the 0731 checkpoint just being released, I am incredibly excited about hosting that model faster than my mmap-ed solution can go. To be honest, I fell back to that after testing GLM 5.2. At roughly 2x the speed, it’s just more useful to me at the moment.

Gefällt dir, wie wir über KI denken?

Diese drei Produkte sind genau aus diesem Denken heraus entstanden. Jedes davon aus einem direkten Kunden-Need heraus, nicht aus einer Pitch-Deck-Session. Wenn eines davon zu deiner aktuellen Baustelle passt: Ein Gespräch. Kostenlos. Kein Commitment.

Dein Kontakt zu uns

Avatar von Simon Huemmer

Dein Thema?

Das Thema interessiert dich? Wenn Du fragen hast, dann melde dich ganz unverbindlich bei uns!

Wie dürfen wir Dich ansprechen?

Höflichkeit ist uns wichtig!

Wie können wir Dich erreichen?

Womit können wir behilflich sein?


Kommentare

Schreibe einen Kommentar

Deine E-Mail-Adresse wird nicht veröffentlicht. Erforderliche Felder sind mit * markiert

Für das Handling unseres Newsletters nutzen wir den Dienst HubSpot. Mehr Informationen, insbesondere auch zu Deinem Widerrufsrecht, kannst Du jederzeit unserer Datenschutzerklärung entnehmen.