FreeToken Unlocks Frontier MoE Inference on Consumer Hardware via Dynamic Co-Execution
AI-summarised brief · reviewed before publication
Researchers from UC Berkeley and MIT have released FreeToken, an open‑source inference engine that enables frontier Mixture‑of‑Experts (MoE) models to run on consumer‑grade hardware. Co‑authored by Databricks founders Matei Zaharia and Ion Stoica, the system replaces static expert offloading with a dynamic co‑scheduling policy (q), splitting token computation between CPU cores and GPU tensor cores based on real‑time PCIe bandwidth. By using a fast weight format, full‑layer double buffering, and an elastic memory manager that reallocates VRAM on the fly, FreeToken overlaps weight streaming with active computation, eliminating stalls caused by cache misses. The approach treats personal machines as elastic, heterogeneous fabrics rather than constrained datacenter nodes, aiming to bring large‑scale sparse AI capabilities to edge devices.
💡 Why It Matters
- · It demonstrates a practical path for deploying billion‑parameter sparse models on everyday PCs, potentially democratizing access to cutting‑edge AI without costly infrastructure.
