Training on Apple Neural Engine with Private APIs

The ANE ↗ project by maderix ↗ is a breakthrough research proof-of-concept: training neural networks directly on Apple’s Neural Engine (ANE) via reverse-engineered private APIs — without CoreML, Metal, or GPU.

The M4 ANE delivers 15.8 TFLOPS FP16 (18.6 TOPS measured), yet Apple restricts it to inference-only through CoreML. This project breaks through that barrier, proving the limitation is software support, not hardware capability.

Software Stack#

CoreML is not the only path in. The _ANEClient class in AppleNeuralEngine.framework provides direct access to the compile → load → evaluate pipeline:

id client = [_ANEClient sharedConnection];
id model = [_ANEModel modelAtURL:compiledURL key:@"mykey"];

[client compileModel:model options:@{
    @"kANEFModelType": @"kANEFModelMIL",
    @"kANEFNetPlistFilenameKey": @"model.mil"
} qos:21 error:&err];

[client loadModel:model options:@{} qos:21 error:&err];

Instead of compiling to .mlmodelc then loading, _ANEInMemoryModelDescriptor compiles MIL text directly in memory — no disk round-trip. This is the key enabler for training.

ANE uses IOSurface for I/O — the same shared memory mechanism as GPU textures, enabling zero-copy GPU↔ANE pipelines.

MIL — The ANE Intermediate Language#

ANE doesn’t accept ONNX or protobuf. It uses MIL (Model Intermediate Language) — an SSA representation with explicit types and shapes:

program(1.3) {
    func main<ios18>(
        tensor<fp16, [1, 1024, 1, 1024]> x,
        tensor<fp16, [1, 1024, 1, 1024]> w
    ) {
        tensor<fp16, [1, 1024, 1, 1024]> out =
            matmul(transpose_x = false, transpose_y = false,
                   x = x, y = w);
    } -> (out);
}

Tensor layout follows NCDHW + Interleave: [Batch, Channels, Depth, Height, Width]. A 1024×1024 matrix becomes [1, 1024, 1, 1024] in 4D.

Training Architecture#

The dynamic pipeline uses shared ANE kernels with weights packed into the spatial dimension — no recompilation when weights change.

MHA models (Stories110M) — 6 kernels per layer:

GQA models (Qwen3-0.6B) — 10 kernels per layer with separate woFwd, qBwd, kvBwd for grouped-query attention.

CPU handles: RMSNorm forward/backward, residual connections (DeepNet α scaling), loss computation, dW gradient accumulation (cblas_sgemm), Adam optimizer.

Performance Results#

Training throughput (M4):

INT8 W8A8 quantization — 1.88x speedup:

INT8 activations halve L2 SRAM bandwidth between tiles via quantize/dequantize.

Key Optimizations#

• Channel-first CPU layout — matches IOSurface [1,C,1,S] format, eliminates all transpose overhead
• vDSP vectorized RMSNorm — 10x faster (6.7ms → 0.7ms)
• GCD async cblas overlap — dW gradient sgemms run in parallel with ANE evals
• Deferred cblas wait — pushes wait into the next step’s forward pass
• ANE RMSNorm fusion — RMSNorm folded into forward kernels via MIL ops
• Forward taps — Q, K, V, attention scores exposed via concat outputs, avoiding CPU recompute
• exec() restart — bypasses ~119 ANE compile limit per process

Limitations#

• SDPA causal masking — ANE hardware ignores attn_mask; causal attention decomposed into Q@K^T (ANE) → mask+softmax (CPU) → scores@V (ANE)
• ~119 compile limit — compiler leaks resources; workaround via exec() restart with checkpoint
• FP16 gradient underflow — backward matmuls underflow in fp16; fixed with global loss scaling
• Low utilization — only ~5-9% of peak, many element-wise ops fall back to CPU

References#

• GitHub — maderix/ANE ↗
• Inside the M4 Apple Neural Engine, Part 1: Reverse Engineering ↗
• Inside the M4 Apple Neural Engine, Part 2: Benchmarks ↗
• Inside the M4 Apple Neural Engine, Part 3: Training ↗
• hollance/neural-engine ↗ — Community ANE documentation
• apple/ml-ane-transformers ↗ — Apple’s reference transformers for ANE