Crossfire-X
Heterogeneous Distributed LLM Inference · ANE + CUDA + Metal
What It Is
A distributed inference pipeline that lights up five compute targets inside a single EXO-orchestrated cluster: NVIDIA CUDA (RTX 5090) for prefill, Apple Metal for decode, the Apple Neural Engine for speculative-decode drafts, CPU for KV cache management, and NVMe for Flash-MoE expert streaming. Composes TriAttention KV compression with TurboQuant+ weight compression so a consumer WiFi link stops being the bottleneck.
Every Mac in an EXO cluster leaves ~19 TFLOPS of ANE compute dark, and MoE models that exceed node memory are completely unservable without expert streaming. Crossfire-X lights up both — then proves composed compression makes a common consumer interconnect (WiFi) good enough so TB4/USB4 becomes optional, not load-bearing.
What I Built
- Seven execution policies (P0–P6): single-node fallback, EXO baseline, ANE speculative decode (E2B draft), TQ4_1S compressed weights, TriAttention KV path, full-stack stacked, and Flash-MoE expert streaming for models exceeding node memory
- AutoPilot policy engine — deterministic decision tree with optional UCB1 / Thompson bandit routing across the seven policies
- C0–C7 ablation matrix isolating each component (single-node baseline → EXO → ANE draft → KV compression → weight compression → Flash-MoE)
- Hardware bring-up: RTX 5090 (32GB) + Apple M4 Max (64GB unified, 16-core ANE) + NVMe — five targets, one pipeline
- Compression stack: TriAttention (~10.7× KV reduction) composed with TurboQuant+ for ~6.8× total KV transfer reduction across nodes
- Target model family: Gemma 4 31B dense, Gemma 4 26B-A4B MoE, Gemma 4 E2B for ANE draft, plus Orion Forge fused specialist
Tech Stack
Active integration across Tier 0 (EXO baseline) → Tier 1 (Flash-MoE) → Tier 2 (ANE + composed compression) → Tier 3 (AutoPilot routing). Apache 2.0. Builds directly on Orion (ANE programming) and integrates community work from EXO Labs, ANEMLL, and TurboQuant+.