Running large language models locally often demands substantial RAM capacity. For models like Gemma 4 26B-A4B, loading all weights into system memory can require 16GB to 32GB RAM. TurboFieldfare overcomes this limitation by leveraging Mixture-of-Experts (MoE) architecture to run a 26-billion parameter model using only approximately 2GB RAM.
The RAM Bottleneck of Large LLMs#
Popular inference frameworks such as llama.cpp or MLX typically load all model parameters into system memory or VRAM. On Macs with 8GB unified memory (such as an M2 MacBook Air), executing models larger than 7B parameters often leads to memory exhaustion or severe disk swap overhead.
[!NOTE] Mixture-of-Experts (MoE) architectures activate only a few designated experts for each output token rather than evaluating every parameter simultaneously.
How TurboFieldfare Streaming Works#
TurboFieldfare partitions the model into two primary components:
- Shared Core & KV Cache (~1.35 GB): Kept permanently resident in RAM for core computations and conversation state retention.
- Dynamic Experts (SSD On-Demand Streaming): Expert layers remain stored on high-speed NVMe SSDs and are streamed into memory precisely when required by the routing layers.
// Concept simulation of on-demand SSD expert loading
func loadExpertOnDemand(expertId: Int) async throws -> MetalBuffer {
if let cached = cache.get(expertId) {
return cached
}
let buffer = try await ssdReader.readExpertChunk(id: expertId)
cache.insert(expertId, buffer)
return buffer
}[!TIP] By leveraging Apple Silicon NVMe disk read speeds, streaming parameter chunks on demand achieves generation speeds of 5–6 tokens/second on an M2 Air and 30+ tokens/second on M5 Pro.
References#
- Official Repository: drumih/turbo-fieldfare ↗
- Community TurboFieldfare performance records and benchmarks.