Orion: Programming Apple's Neural Engine for LLM Training (Not Just Inference)

A few people asked me to write up the Orion work the same way I wrote up KALAVAI — without the abstract, without the API listings, without the MIL IR fragments. This is my honest attempt.

Every iPhone, iPad, and Mac sold since 2020 ships with a piece of silicon that almost no one has ever talked to. Apple calls it the Neural Engine. The hardware press calls it an NPU. Internally it's known as the ANE. There are now well over two billion of these chips in the world. Each one delivers somewhere around 19 TFLOPS of fp16 compute — Apple's marketing claims 38 TOPS in INT8, though maderix's reverse-engineering work shows the ANE dequantizes to fp16 before compute, so the honest number is the lower one. It's a real accelerator. It sits next to the GPU. It is faster, per watt, for the operations it supports than anything else on the chip.

And, with vanishingly few exceptions, it has been dark.

The reason it's dark is that Apple ships exactly one public way to use it: a framework called CoreML. CoreML is a perfectly reasonable framework for what it does, which is run pre-baked inference models on devices where Apple has decided what those models can be. It has a list of supported operators. It has a model format. It has a compiler that does conversions from PyTorch and TensorFlow. What it does not have — what it has never had — is a way to train. There is no backward pass in CoreML. There is no on-device fine-tuning. There is no path from "I have a model and I want to specialise it to my user's data" to "the ANE handled the forward and backward passes."

If you wanted to train on Apple Silicon, your options were Metal — using the GPU — or the CPU, or the cloud. The ANE, the dedicated piece of silicon Apple designed for neural-network workloads, was off-limits by design.

Orion is what happened when I decided that wasn't an acceptable answer.

What Orion is

Orion is a local AI runtime that talks to the Apple Neural Engine directly, through Apple's private frameworks — _ANEClient, _ANECompiler, the MIL intermediate representation that underlies CoreML — without going through CoreML itself. It compiles its own MIL programs. It manages its own memory. It schedules its own kernels. It does forward passes on the ANE. It does backward passes on the ANE. It updates weights on the CPU and reloads them onto the ANE without recompiling anything. It supports LoRA-style adapter hot-swap at inference time. It ships as a single binary that builds with clang && make. No Xcode. No CoreML. No cloud.

┌─────────────────────────────────────────────────────────────┐
│  Prompt → Tokenizer → ANE Runtime → CPU Sampling → Output  │
│                                                             │
│  170+ tok/s inference  ·  3.8× faster training via delta    │
│  compile  ·  LoRA hot-swap without recompilation            │
└─────────────────────────────────────────────────────────────┘

The paper that documents the system — Orion: Characterizing and Programming Apple's Neural Engine for LLM Training and Inference, arXiv 2603.06728 — has three contributions. First, a catalogue of twenty ANE constraints, fourteen of which were previously undocumented anywhere outside Apple. Second, a compiler and runtime that turn a graph-IR description of a transformer into ANE-executable MIL programs with five optimisation passes. Third, a training-time optimisation called delta compilation that reduces per-step recompilation from 4,200 milliseconds to 494 milliseconds — an 8.5× recompilation speedup that translates into a 3.8× end-to-end training speedup.

I'll come back to each of these. But I want to start with the part that surprised me most, because it's the part that explains why the rest of the work was even possible.

The 119-compile limit, and why training looked impossible at first

When you ask the ANE to run a new program — a new fused chunk of compute, like "the forward pass of this transformer layer with these specific weights" — Apple's runtime allocates a model descriptor, compiles MIL to ANE microcode, and registers it with the kernel. This is the cost of admission. It is not cheap. For a 110-million-parameter transformer with sixty weight-bearing kernels, the per-step compile cost is roughly 4.2 seconds.

That, by itself, is bad — but survivable. If every training step costs 4.2 seconds in compilation plus a few hundred milliseconds in compute, you can still train, you just train slowly. So when I started the training work, I was reconciled to that. I figured we'd squeeze the compile cost down later.

What I didn't expect, and what almost killed the project, was a much harder ceiling: Apple's runtime limits each process to approximately 119 compilations before it refuses to compile any more. Once you hit that ceiling, the runtime returns an error and you cannot recompile anything until you re-launch the entire process.

For inference, 119 compiles is more than enough. You compile the model once at startup, run it forever, never recompile. For training, 119 compiles is a guillotine. A naive training loop recompiles every step (new weights, new MIL programs, new model descriptors). At 60 kernels per step and a 119-compile budget, that gets you exactly one training step per process. After that, you exec() the entire training binary to get a fresh budget, reload your checkpoint, and run one more step. The original v1.0 of Orion did exactly this — one step per process, an exec() restart between every step. A 1,000-step training run took 85 minutes of wall time, and most of that 85 minutes was process launch overhead.

That was clearly not going to work. The whole point of Orion was to make on-device training feasible, and "you have to restart your binary between every gradient step" is not feasible.

The fix is the part I'm proudest of in the whole project, because the answer turned out to be much smaller than I expected.

Delta compilation, or: don't recompile, just rewrite the weights

The realisation that broke the problem open was this: the ANE's compile step does two things, and only one of them is expensive. The expensive part is turning MIL text into ANE microcode — taking a graph description and lowering it through the optimiser into actual instructions for the silicon. The cheap part is binding weight tensors to that compiled program. If the structure of the program doesn't change between training steps — same architecture, same layers, same dimensions, just different weight values — then in principle you should not need to recompile the structure. You just need to swap the weights.

Apple's runtime did not expose this. There was no public "reload weights" call. But it turned out, after enough time spent reading lldb output and tracing what the framework was actually doing, that the underlying machinery to swap weights did exist in the private API surface. You could unload an existing ANE program, update the weight BLOBFILE on disk that the program had been compiled against, and reload the program — bypassing ANECCompile() entirely. The reload is roughly 494 milliseconds for sixty kernels, versus the 4,200 milliseconds the full recompile would cost. That's the 8.5× speedup.

But the more important consequence is what it does to the compile budget. Because the reload path doesn't count against the 119-compile limit, a training loop using delta compilation needs to compile each ANE program exactly once, ever — at process startup. Sixty kernels, about 4.5 seconds of one-time compile cost, and then every subsequent training step is just forward → loss → backward → weight update → delta reload. No exec() restarts. No process boundaries. No compile-budget management. A 1,000-step training run that used to take 85 minutes and 1,000 process launches now takes 22.4 minutes in a single process, with zero exec()s.

That's the whole engineering trick. It looks small written down. It took weeks to find.

The 22-minute number is the one I quote when people ask whether on-device training is real. It is real. The 1,000-step run hit a loss curve from 11.83 to 8.9, with zero NaN occurrences across all 1,000 steps, no memory leak, ten clean checkpoint saves, and a single process for the entire run.

Twenty constraints, fourteen of them new

The other thing the project produced, almost as a byproduct, was a catalogue of ANE constraints. Some of these are documented somewhere in the depths of Apple's developer forums or were found by maderix during the original reverse-engineering work that Orion builds on. Most of them are not documented anywhere I can find. They are real constraints — if you violate them, your MIL program fails to compile, or it compiles but crashes at evaluation, or it runs but returns garbage. I wrote them down because I wished someone had written them down for me.

A few of the ones I keep needing to explain:

The ANE rejects the concat MIL op. You can express concatenation with a multi-output program — a single MIL program that produces multiple output tensors — but the explicit concat op is not in the ANE's supported set. The first time you discover this, it is by writing a perfectly valid MIL program with a concat in it, hitting an opaque compile error, and spending three hours staring at it.

gelu is not a valid MIL operator. You decompose it into the tanh approximation: 0.5 × x × (1 + tanh(√(2/π) × (x + 0.044715 × x³))). Some transformers use exact GELU. Some use the approximation. On the ANE, you must use the approximation, full stop.

Multi-output, multi-input programs require uniform IOSurface allocation sizes. If you have a program that takes three inputs and produces four outputs, all seven of those IOSurface buffers must be allocated to the same size — the maximum of the actual sizes needed. Anything else and the program silently produces wrong outputs. This one took the longest to find because it doesn't crash. It just lies.

The weight dict must be @{}, the empty dictionary, not nil. For programs that don't have weights — pure compute programs, for example, like a softmax kernel that takes one tensor and produces another — Apple's runtime expects an empty Objective-C dictionary, not a null pointer. Passing nil produces an undocumented error code. Passing @{} works.

Minimum IOSurface allocation is approximately 49KB. Programs compile fine with seq_len=1 (which would naively need a much smaller buffer), but they fail at evaluation time. You pad to the minimum allocation, even if you're throwing most of the buffer away.

There are fifteen more like these in the paper. The full catalogue lives in docs/ane_constraints.md in the repo. If you are trying to do any non-trivial work against the ANE, that file is the single most useful thing I can give you. It is the document I wish had existed when I started.

The NaN-fix story, or: the only thing harder than getting training to work is getting it to stay working

Training works. Training continued to work for the first 200 steps of every 1,000-step run, reliably. Training then started producing NaN losses, and would NaN for the rest of the run.

There turned out to be three separate bugs causing NaN cascades, and they took longer to find than the delta-compile work did. I want to mention them briefly because they are the kind of bugs that only show up when you're doing things that the framework's authors didn't expect you to do.

Bug one: stale ANE programs after checkpoint resume. If you compiled the ANE programs at startup and then loaded checkpoint weights, the programs were compiled against the original (random) weights — but training was using the new checkpoint weights. The forward pass produced output for the old weights, the backward pass produced gradients for the new weights, and the gradient-weight mismatch diverged spectacularly within a few hundred steps. The fix was deferring ANE program compilation until after the checkpoint was loaded. One ordering change, weeks of confusion.

Bug two: fp16 overflow cascades. The ANE runs in fp16. Large activations — common in transformer layers, especially after attention — could overflow to Inf during forward pass. The Inf propagated through softmax (producing 0/0 = NaN) and through cross-entropy (producing NaN losses). The fix was clamping activations to the fp16 range [-65504, 65504] before any operation that could amplify their magnitude — softmax, layer norm. Apple's CoreML framework presumably does something like this internally; bypassing CoreML meant I had to do it myself.

Bug three: a third issue specific to the gradient-accumulation path that I'll spare the details on except to say it involved CPU-side weight buffers being read before the ANE had finished writing them via Grand Central Dispatch's async dispatch queue, and the fix was an explicit barrier before the read. Pure ordering bug. Took two days to isolate.

After all three were fixed, the 1,000-step training run produced zero NaN losses across 1,000 steps, and a 5-chain × 5-step stability stress test produced zero NaN across 25 steps and a cross-chain loss standard deviation of 0.003 at step one and 0.007 at step five — i.e., the seeds converged on each other rather than diverging. The training is stable.

LoRA hot-swap, or: making the ANE useful for personalisation

The other piece of work that mattered to me was making the ANE useful for adapter-based fine-tuning. The motivating use case is concrete: imagine an iPad with a base model and a small set of LoRA adapters — one for legal writing, one for medical, one for translation, one for code — that the user can switch between on the fly. Each adapter is small (a few megabytes), can be downloaded or trained locally, and should activate without restarting the model.

The challenge is that the standard LoRA implementation bakes the adapter weights into the compiled program. That means switching adapters requires recompiling — and we already established that recompiling is expensive, even with delta reload.

The solution was to make the adapter matrices A and B into IOSurface inputs rather than baked weights. The compiler emits MIL that computes Y = conv1x1(x, W_base) + alpha * (x @ A) @ B, with A and B passed as input tensors at evaluation time rather than as compiled-in constants. Switching adapters then becomes a tensor swap, not a recompile. Verified: different adapters produce different outputs with zero recompiles per swap, and the IOSurface dispatch overhead is well under a millisecond. Seventeen out of seventeen tests pass.

The LoRA work also produced three more newly-discovered ANE constraints (numbers 18, 19, and 20 in the catalogue) — which is roughly what I've come to expect from this work. Every non-trivial thing I try to do on the ANE produces at least one new constraint I have to write down.

What I'm not claiming

I want to be explicit about the boundary conditions of this work, because they're real and I would rather state them than let a reader assume more than I'm claiming.

The models are small. GPT-2 124M for inference. Stories-110M for training. Both are well below the size threshold where you can ask "is this useful for what people want from on-device AI in 2026?" The mechanism scales; the demonstration so far does not. Stage 3 of the roadmap is bigger models.

CPU decode beats ANE decode on small models. This surprised me. For GPT-2 124M, CPU decode reaches 283 tokens/second, while ANE full-forward decode reaches 170+ tokens/second. The CPU is faster per-token at this scale because the ANE dispatch overhead per token dominates the actual compute. The ANE shines on prefill (where you're processing many tokens at once and the dispatch cost amortises) and on larger models (where compute dominates dispatch). The ANE-vs-CPU crossover point is a real research question, and Orion is the tool to measure it.

The runtime depends on private Apple APIs. Apple could break these in a future OS update. They could ship CoreML training. They could add formal support for _ANEClient directly. Orion is a research prototype that demonstrates the mechanism, not a product that promises permanent API stability. If you ship something on top of Orion, do so with that risk explicit.

Only M4 Max measured at scale. Inference and training numbers in the paper come from a Mac Studio M4 Max 64GB. The mechanism should work on every Apple Silicon device with an ANE, from M1 onward and from A14 onward, but the systematic per-device benchmark sweep is future work.

No comparison to Metal/MPS so far. A clean head-to-head between the ANE and Metal Performance Shaders on the same model and the same task would tell us when each accelerator is the right choice. Orion doesn't yet ship that comparison. It will.

These are not deflections. They are the honest perimeter of what the paper demonstrates. Future work tightens each of them.

What this makes possible

I want to spell out, briefly, the use cases that motivate the work — not because I expect every one of them to materialise in the next year, but because the shape of what becomes possible is the part I keep coming back to.

A local AI assistant on a Mac that genuinely never makes a network call — not for fine-tuning on your own writing style, not for personalisation, not for adapter switching, not for inference. The ANE handles the forward pass; the CPU handles sampling; nothing leaves the device.

A toy robot for kids, running a small LLM on an embedded iPad. Today, that robot is either dumb or it's a privacy hazard. With Orion-style on-device training, it can be smart and keep the child's voice data on the device.

A coding copilot that fine-tunes itself, locally, on your codebase — without any of your code ever crossing the network boundary. The current generation of copilots all assume cloud inference. There is no architectural reason that has to be the case for personalisation.

Medical, legal, and other privacy-sensitive deployments where the data must never leave the device. These exist today only as inference deployments using CoreML or llama.cpp on CPU. On-device fine-tuning, where the model adapts to the specific institution's documents, has been blocked by the absence of a training path on the dedicated accelerator. Orion removes that block.

And — the part I personally care about — ML research that can profile, benchmark, and characterise the ANE as a research target. The ANE is one of the most widely-deployed ML accelerators in the world, and until now there has been no public toolchain for measuring its actual behaviour. The Orion benchmark suite — orion bench kernels, orion bench inference, orion bench training — turns the ANE from a black box into a measurable research subject. The 20 constraints in the paper are the first downstream artifact of that research path. There will be many more.

What Orion is really about

If KALAVAI is about disciplined orchestration of training across many participants, Orion is about disciplined orchestration of a single device's hidden hardware. The thesis is the same. Disciplined engineering beats waiting for a vendor to ship the feature you need.

Apple shipped two billion of these chips. They shipped one public framework that handles a fraction of what the chip can do. The gap between "what the hardware is capable of" and "what the public API exposes" is the engineering opportunity. Orion takes that opportunity by treating the ANE as what it is — a programmable accelerator with documented and undocumented constraints — and building a compiler, runtime, and training loop on top of the private API surface that Apple already ships.

The paper is on arXiv. The code is on GitHub, MIT-style permissive, no Xcode required to build. The 1,000-step training run reproduces in 22 minutes on any M4 Mac with 16GB of memory or more. The benchmark suite runs in minutes and tells you exactly how your ANE compares to mine. The constraint catalogue is one markdown file you can read in fifteen minutes that will save you fifty hours of debugging if you decide to do any non-trivial work against the ANE yourself.

That is the part that I think matters. Most of what I built here is documentation — a catalogue of constraints, a compiler that produces conforming MIL, a runtime that handles the lifecycle, a benchmark suite that quantifies the behaviour. The runtime is interesting, but the documentation is what unlocks downstream work. If you are an ML researcher with a Mac, you can now profile the ANE. If you are a product team trying to do on-device fine-tuning, you can now ship it. If you are a student trying to learn how neural accelerators actually work, you can now read the constraints document and understand what kind of object you are dealing with.

The work is small. The hardware is enormous and entirely public. The combination is the part I'm interested in.

Resources

• Paper: arxiv.org/abs/2603.06728
• GitHub: mechramc/Orion
• ANE constraint catalogue: docs/ane_constraints.md
• Benchmark results: RESULTS.md
• License: Apache 2.0

Orion builds directly on the foundational reverse-engineering work of maderix (private API discovery, hardware characterisation) and ANEgpt (training kernel structure). The compiler, runtime, training loop, delta-compilation pipeline, LoRA adapter-as-input path, and the catalogue of fourteen newly-documented constraints are the Orion contributions on top of that base.