🔍 Read the full analysis: The 32GB RAM Challenge: MoE Inference At 180B Parameters With Transformers on ThorstenMeyerAI.com
Get the latest gadgets delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
A Hugging Face developer proposed an experimental Transformers option that streams selected mixture-of-experts weights from storage into a bounded device cache. In a reported OLMoE test, it generated 34.6 tokens per second versus 38.1 for the standard Transformers path; the proposed 180B model test on a 32 GB Mac has not been reported.
A Hugging Face developer has proposed an experimental change to Transformers that loads selected mixture-of-experts (MoE) weights from storage into a bounded device cache, rather than keeping every expert in fast memory, as described in the original analysis. In a reported test on OLMoE-1B-7B, the method reached 34.6 tokens per second, compared with 38.1 for the standard Transformers path; results for the motivating 180B-model test on a 32 GB Mac have not been provided.
The experimental approach targets MoE models, which route each token through only a subset of a model’s experts. The developer says experts make up more than 90% of the weights in the model discussed, creating the possibility of keeping selected experts near the processor and fetching others only when needed. The implementation is described as using a shared, preallocated device tensor for cached experts, with a table mapping expert IDs to cache slots.
An initial version ran experts one at a time in a Python loop. The developer reports it generated 10.1 tokens per second even when experts were cached, compared with about 32 for vanilla Transformers. The revised cache allowed the existing grouped matrix multiplication kernel to process selected experts together. In the reported OLMoE test, that version reached 34.6 tokens per second, against 38.1 for the baseline.
The developer says the change was submitted to Transformers’ from_pretrained flow as pull request #50717. The source also points to a Hugging Face Space, tarekziade/moe-expert-streaming, with project code. It describes work involving Transformers, a model-specific package and Metal kernels, and says AI tools were used in building the code.
Streaming MoE Experts With Limited Memory
An experimental Transformers change streams selected experts from SSD into a bounded cache. The reported OLMoE test ran close to vanilla speed, but the 180B model on a 32 GB Mac remains untested in the source.
How the cache works
The router selects experts for each token; the experiment fetches those experts into a shared cache.
Choose a subset of experts for the token.
Stream them from storage into available device cache slots.
Use Transformers’ grouped matrix multiplication kernel.
What the test shows
On OLMoE-1B-7B, the shared cache kept the reported speed near the standard path.
- Python loop ran experts one at a time.
- Reached 10.1 tokens per second, even with experts cached.
- Preallocated tensor mapped expert IDs to cache slots.
- Reached 34.6 tokens per second versus 38.1 baseline.
The larger device test is open
The 180B model on a 32 GB Mac is a motivating target, not a measured result in the supplied source.
Promising engineering
The cache design brought the reported small-model test close to vanilla Transformers speed.
SSD performance
Storage traffic and cache misses could change speed when experts must be fetched repeatedly.
Measure the target setup
Report speed and memory use on the larger model, with storage, cache settings and workload specified.
Lower Memory Use, Speed Trade-Off
If the method performs reliably on models whose expert weights exceed available memory, it could make some larger MoE models usable on hardware that cannot hold all their weights at once. That would be relevant to developers seeking local inference on laptops or single-GPU systems, where memory capacity can prevent a model from running even though each token uses only some experts.
The reported result addresses a narrow but relevant question: whether expert caching can avoid the slowdown of processing experts individually in a Python loop. The revised design came close to the reported in-memory Transformers speed on OLMoE. But fetching experts from storage adds data movement, and the test does not establish how that cost affects generation when the model’s working set does not fit in memory.
The source also reports comparisons on Qwen3.6-35B-A3B using an A10G GPU with 24 GB of memory and ample system RAM. It says Strata generated 82–88 tokens per second with Q8_0, while llama.cpp with Q8_0 and its fit option reached 63–64. Those figures describe that model and setup; they are not results for the new Transformers implementation or a 32 GB Mac.
Products Worth Considering

Jackery Explorer 300 Portable Power Station, 292Wh

LG UltraGear 27-inch QHD OLED Gaming Monitor, 240Hz, 0.03ms, G-SYNC Compatible, FreeSync Premium Pro, HDR True Black 400

Bose QuietComfort Ultra Wireless Earbuds (2nd Gen) with Deep Plum Finish and 20W Dual-Port Charger
As an affiliate, we earn on qualifying purchases.
How the Expert Cache Works
MoE models contain multiple specialist subnetworks, called experts, and route each token through a selected subset. That design can reduce the computation required per token compared with activating every model parameter, but the model’s full set of weights still has to be stored somewhere. When the weights exceed device memory, inference systems must use system RAM or storage, move weights between memory tiers, or rely on other offloading strategies.
The source describes the proposed Transformers option as fetching routed experts into a bounded cache while retaining grouped expert computation. It contrasts this with approaches it says can run experts on the CPU, move offloaded weights over PCIe, or transfer an entire layer’s expert bank before processing. These are the source author’s characterizations, and the reported comparisons use a different model and hardware setup from the OLMoE test.
The practical target named in the report is running a large MoE model, described as a 180B-parameter use case, on a Mac with 32 GB of unified memory. That is presented as a goal motivating the experiment, not as a completed benchmark result.
“The experimental path reached 34.6 tokens per second, compared with 38.1 for vanilla Transformers.”
— The developer’s reported OLMoE benchmark
The 32 GB Mac Test Is Missing
The source gives no measured result for the 180B model on a 32 GB Mac. It also does not specify the storage device, cache size, prompt conditions or repeated-run results for the OLMoE comparison, making it difficult to judge how performance may vary across workloads.
The status of pull request #50717 is not stated: it is unclear whether the change has been accepted, merged or released. The reported tests also do not establish how the cache performs when routing patterns change quickly, or how the proposed implementation compares with Strata on the same model and hardware. The available evidence supports a result on the reported OLMoE test, not a general claim about large-model inference from SSD.
Pull Request and Larger-Model Results
The next development to watch is whether Transformers pull request #50717 is accepted and made available in a release. The linked Hugging Face Space gives readers a place to inspect the experimental project, according to the source.
To evaluate the motivating use case, further tests would need to report generation speed and memory use for the 180B MoE model on a 32 GB Mac, along with the storage device, cache settings and workload. A comparison against alternatives on the same model and hardware would help show how much performance the approach retains when expert weights must actually be fetched from storage.
Key Questions
What did the Transformers experiment change?
It proposed loading routed MoE experts from storage into a bounded device cache, then using grouped matrix multiplication to process selected experts together.
What speed did the reported test reach?
On OLMoE-1B-7B, the developer reports 34.6 tokens per second, compared with 38.1 for the standard Transformers path. The source does not provide enough test details to establish how the result varies across workloads.
Was a 180B model tested on a 32 GB Mac?
No such benchmark result is reported. The 180B model and 32 GB Mac are described as a motivating target, not a completed test.
Is the change available in Transformers now?
The developer says it was proposed in pull request #50717. The source does not say whether the pull request was merged or released.
Primary source: Hugging Face · via ThorstenMeyerAI.com
