Wiring MLX and Core ML ANE Pipelines in Swift 6: On-Device Inference Without the Latency Cliff

What We Are Building

By the end of this tutorial, you will have a production-ready Swift 6 actor topology that routes on-device inference requests between MLX's GPU compute and Core ML's ANE scheduler - without ever blocking the main actor. We will wire up two isolated actors, a coordinator with runtime routing logic, and validate the setup against real benchmarks on M-series chips. Target outcome: sub-50ms first-token latency on quantized 3B models. That is achievable, and here is the architecture that gets you there.

Prerequisites

• Xcode 16+ with Swift 6 mode enabled
• macOS 15+ running on M-series hardware (M1 or later)
• Familiarity with Swift's async/await and basic actor concepts
• MLX Swift package (github.com/ml-explore/mlx-swift)
• A Core ML compiled model (.mlmodelc) - Llama 3.2 3B 4-bit AWQ works well as a reference

Step 1 - One Actor Per Compute Backend

Let me show you a pattern I use in every project: strict actor isolation, one backend per actor, zero shared mutable state. Most teams treat inference like a network call - async/await, fire and forget, handle on the main actor. That mental model breaks the moment your model is 2GB and the ANE scheduler starts competing with Core Animation for memory bandwidth.

Swift 6 makes data races a compile error, which forces the architectural conversation you should have had earlier. You cannot share an MLModel instance across actor boundaries without explicit sendability guarantees.

Here is the minimal setup to get this working:

// MLX GPU actor - owns the model graph and KV cache
@globalActor actor MLXInferenceActor {
    static let shared = MLXInferenceActor()
    private var model: LLMModel?
    private var kvCache: KVCache?

    func load(_ config: ModelConfig) async throws {
        model = try await LLMModel.load(config)
        kvCache = KVCache(capacity: config.contextLength)
    }

    func generate(prompt: MLXArray, maxTokens: Int) -> AsyncStream {
        AsyncStream { continuation in
            Task {
                guard let model else { continuation.finish(); return }
                for token in model.stream(prompt, cache: kvCache) {
                    continuation.yield(token)
                }
                continuation.finish()
            }
        }
    }
}

// Core ML ANE actor - owns the compiled model and prediction context
@globalActor actor ANEInferenceActor {
    static let shared = ANEInferenceActor()
    private var compiledModel: MLModel?

    func load(url: URL) async throws {
        let config = MLModelConfiguration()
        config.computeUnits = .cpuAndNeuralEngine // ANE-first, CPU fallback
        compiledModel = try MLModel(contentsOf: url, configuration: config)
    }

    func predict(input: MLFeatureProvider) async throws -> MLFeatureProvider {
        guard let compiledModel else { throw InferenceError.notLoaded }
        return try compiledModel.prediction(from: input)
    }

    func stream(_ request: InferenceRequest) -> AsyncStream<Token> {
        AsyncStream { continuation in
            Task {
                do {
                    let output = try await predict(input: request.toMLFeatureProvider())
                    for token in output.toTokenSequence() {
                        continuation.yield(token)
                    }
                } catch { }
                continuation.finish()
            }
        }
    }
}

Neither actor touches MainActor. Your UI layer subscribes to AsyncStream<Token> and updates SwiftUI state via @MainActor task groups. The inference path never blocks the run loop.

Step 2 - Know Which Backend to Route To

Here is the benchmark data from Llama 3.2 3B (4-bit AWQ, ~1.8GB on-disk) on an M3 Pro with 18GB unified memory, macOS 15.2. Medians from 20 runs, first two cold-start runs excluded:

Rule of thumb: if your p95 context length stays under 1K tokens and you are serving one request at a time, route to ANE. If you are building a document summarization pipeline or supporting concurrent requests, MLX on GPU gives you the throughput.

Step 3 - Wire the Coordinator

actor InferenceCoordinator {
    private let aneActor = ANEInferenceActor.shared
    private let mlxActor = MLXInferenceActor.shared

    func route(request: InferenceRequest) async -> AsyncStream<Token> {
        switch request.contextLength {
        case ..<1024 where !request.isBatched:
            return await aneActor.stream(request)
        default:
            return await mlxActor.generate(
                prompt: request.toMLXArray(),
                maxTokens: request.maxTokens
            )
        }
    }
}

Changing the cutoff threshold is a one-line change, not a refactor. All routing logic lives here and nowhere else.

Gotchas

The docs do not mention this, but Core ML's ANE scheduler silently falls back to CPU for unsupported operator types. These four account for the majority of unexpected CPU fallback in transformer pipelines:

• LayerNorm with dynamic sequence length - most frequent culprit; static-shape variants compile cleanly
• RoPE with variable context - dynamic sequence indexing breaks ANE dispatch
• Grouped-query attention (GQA) - multi-head key/value grouping patterns are not fully supported on current ANE hardware
• Custom activation functions - anything beyond ReLU, GELU, and Swish typically falls back to CPU

Run mlmodelc compilation with --compute-units cpuAndNeuralEngine and inspect the compilation report. Any op marked cpu_only is killing your ANE gains. If more than 15% of ops fall back to CPU - especially LayerNorm, RoPE, or GQA - your model architecture needs modification before ANE routing delivers real gains.

Also: MLX GPU and Core ML ANE have incompatible threading models. Mixing them on a shared actor is a deadlock waiting to happen. Swift 6 enforces the boundary at compile time - use it.

Conclusion

One actor per compute backend. Compile the Core ML model and audit the ops report. Let context length and batch size drive routing, not intuition. Build a coordinator that makes the routing decision at runtime based on request shape, and benchmark against your actual p95 context distribution before picking a fixed cutoff. The architecture here gets you to sub-50ms first-token latency - but only if you do the operator audit first.