Research
What It Is
The first open end-to-end system for direct ANE inference and training — bypassing CoreML to program Apple's Neural Engine directly.
The Problem
Apple's Neural Engine delivers extraordinary on-device ML performance, but it's a black box. No public API, no documentation for direct access. CoreML adds latency overhead. This paper opens the ANE.
Approach
- Reverse-engineered ANE instruction set and memory layout from hardware traces
- Built a compiler pipeline: MLIR → ANE binaries without CoreML intermediary
- Implemented delta compilation — only recompile changed model segments
- Designed a local-first fallback routing system (ANE → GPU → CPU)
Results
- 8.5× ANE compilation speedup via delta compilation
- Direct ANE execution without CoreML overhead
- First open system enabling ANE training (not just inference)
Stack
MLIRMetalSwiftPythonApple Neural EngineCoreML internals