TurboFieldfare (Part 2): Swift & Metal Runtime Optimization

Achieving real-time streaming performance from SSD requires precise hardware control. Rather than relying on existing frameworks like MLX or llama.cpp, TurboFieldfare implements a custom inference runtime crafted natively with Swift 6.2 and Metal Compute Shaders.

Why Build a Custom Engine?#

Deep macOS system integration requires raw control over memory caching and low-level disk I/O pipelines:

• Swift Concurrency: Manages asynchronous SSD disk reads in parallel with GPU compute pipeline dispatching.
• Metal Kernel Shaders: Executes high-throughput matrix multiplications and MoE activation layers directly on Apple Silicon GPU cores.

#include <metal_stdlib>
using namespace metal;

kernel void moe_expert_compute(
    device const float* input [[buffer(0)]],
    device const float* expert_weights [[buffer(1)]],
    device float* output [[buffer(2)]],
    uint id [[thread_position_in_grid]]
) {
    // Matrix computation for loaded expert weights
    output[id] = input[id] * expert_weights[id];
}

Mitigating SSD Disk Latency#

Disk read access latency poses the primary bottleneck in weight streaming. TurboFieldfare resolves this via three core techniques:

1. LFU (Least Frequently Used) Caching: Retains high-frequency “hot experts” inside the compact ~2GB memory budget.
2. Chunked Prefill: Subdivides input prompt processing into smaller chunks to streamline sequential I/O throughput.
3. Quantized Weight Formats: Compresses expert weight payloads to minimize raw bytes transferred per token.

[!IMPORTANT] The repository includes over 100 documented performance experiments evaluating caching algorithms, Metal kernels, and I/O access patterns.

References#

• Swift & Metal Source Code: drumih/turbo-fieldfare ↗
• Apple Developer Metal Shading Language Documentation.