On-Device Inference with GGUF, MLX, and ONNX Runtime

Running Large Language Models (LLMs) on edge devices—smartphones, tablets, and embedded Linux boards—requires navigating a fragmented landscape of hardware accelerators. Unlike server-side inference, where massive GPU clusters dominate, edge inference is constrained by thermal limits, battery life, and memory bandwidth. The three dominant frameworks today—GGUF, Apple’s MLX, and Microsoft’s ONNX Runtime—represent distinct approaches to solving the same problem: how to move tensors from memory to compute units with minimal overhead.

The GGUF Ecosystem: Universal Compatibility via Quantization

GGUF (GPT-Generated Unified Format) is the successor to GGML, designed primarily by Georgi Gerganov for the llama.cpp inference engine. Its primary mechanism is aggressive quantization. While models are often trained in FP16 or BF16, gguf allows storage in INT4, INT8, or mixed-precision K-quant formats.

The technical advantage of gguf lies in its hardware abstraction. It does not rely on a single accelerator. Instead, it uses a layered execution backend:

1. CPU: Uses SIMD instructions (AVX2, NEON) for baseline execution.
2. Vulkan/Metal: Offloads compute to the GPU via shaders.
3. CUDA/ROCm: For NVIDIA and AMD discrete GPUs (rare on mobile, common on edge PCs).

The bottleneck in gguf inference is almost always memory bandwidth, not compute. Because quantized models (e.g., Q4_K_M) are small, the CPU/GPU can process them faster than they can be fetched from RAM. llama.cpp mitigates this by optimizing KV-cache management and using multi-threading to overlap data loading with computation.

// Pseudo-code: llama.cpp inference loop
llama_context *ctx = llama_init_from_gguf("model.gguf");
llama_batch batch = llama_batch_init(128);
// Add tokens to batch
for (int i = 0; i < tokens.size(); i++) {
    llama_decode(ctx, batch);
    // Process output logits
}

gguf’s strength is universality. A single .gguf file runs on an iPhone, a Raspberry Pi, and a Windows PC with minimal code changes. However, it lacks deep hardware-specific optimizations for NPUs (Neural Processing Units) found in modern mobile chips.

Apple MLX: Unified Memory and Symbolic Graph Construction

If gguf is the "universal translator," MLX is the "native speaker" for Apple Silicon. MLX was built specifically to exploit the unified memory architecture of M-series chips.

In traditional architectures (CPU + separate GPU VRAM), data must be copied between memory pools, incurring latency. In Apple’s unified memory, the CPU, GPU, and Neural Engine share the same physical memory space. MLX leverages this by allowing zero-copy data sharing. A tensor created on the CPU can be passed directly to the GPU kernel without serialization or copying.

Furthermore, MLX employs a symbolic graph construction model. Operations are recorded in a computation graph and executed only when needed, similar to PyTorch’s dynamic graphs but optimized for static-shape efficiency where possible. This allows for dynamic model architectures (like varying sequence lengths) without the overhead of recompiling kernels. This architecture is particularly effective for on device inference scenarios where memory latency is critical.

# MLX Python API example
import mlx.core as mx
import mlx.nn as nn
 
# Define a simple linear layer
linear = nn.Linear(128, 256)
 
# Execute on GPU automatically due to unified memory
x = mx.random.normal((1, 128))
output = linear(x)

mlx is currently the highest-performance option for iOS/macOS devices. It supports advanced quantization schemes (like 4-bit and 8-bit) that integrate tightly with the Apple Neural Engine. However, it is strictly limited to Apple hardware.

ONNX Runtime Mobile: Hardware Delegation

onnx runtime (ORT) takes a different approach: hardware delegation. ONNX is a neutral format for representing machine learning models. ORT acts as the runtime engine that optimizes this graph and delegates specific operations to available hardware providers.

On mobile devices, ORT uses "providers" to offload work:

• CoreML Provider (iOS): Delegates layers to the Apple Neural Engine.
• SNPE (Snapdragon Neural Processing Engine): For Qualcomm Android chips.
• DirectML: For Windows GPU acceleration.

The mechanism here is graph optimization. Before deployment, the model is converted to ONNX format, where tools like onnx-simplifier perform operator fusion and constant folding. At runtime, ORT analyzes the computational graph and maps subgraphs to the most efficient hardware backend. This flexibility makes it a strong candidate for edge ai deployments where diverse hardware capabilities must be supported uniformly.

# Convert PyTorch model to ONNX
python -c "import torch; torch.onnx.export(model, dummy_input, 'model.onnx')"
 
# Run with ONNX Runtime Mobile
import onnxruntime as ort
session = ort.InferenceSession("model.onnx", providers=['CoreMLExecutionProvider'])
results = session.run(None, {"input": input_data})

onnx runtime’s strength is cross-platform consistency. If you need to deploy the same model to Android, iOS, and Windows with similar performance characteristics, ORT is the best choice. However, the conversion process can be fragile, especially for models with complex control flow or dynamic shapes, which are common in modern LLMs.

Comparative Tradeoffs

Choosing between GGUF, MLX, and ONNX Runtime depends on your target hardware and deployment constraints.

Opinion: For iOS/macOS development, MLX is currently the superior choice due to its tight integration with Apple’s hardware and ease of use. For cross-platform edge deployment (Android, Linux, Windows), GGUF remains the most pragmatic solution due to its universal compatibility and mature quantization support. ONNX Runtime is best reserved for scenarios where existing model pipelines are already built around the ONNX ecosystem, or when specific NPU acceleration (like Qualcomm’s SNPE) is required for power efficiency.

Common Pitfalls

1. Incorrect Quantization Mapping: When converting models to gguf, ensure that the quantization scheme matches the target hardware's precision capabilities. Mismatched precision can lead to significant accuracy degradation or runtime errors on devices with limited FP16 support.
2. Dynamic Shape Handling in ONNX: ONNX Runtime can struggle with models that have highly dynamic input shapes (e.g., variable sequence lengths in LLMs). Always use dynamic_axes correctly during export and test thoroughly on the target device, as static shape optimization may fail or produce incorrect results.
3. NPU Driver Availability: Relying on hardware delegation via SNPE or CoreML requires that the specific NPU drivers are installed and accessible on the target device. In development environments or older devices, these providers may fall back to CPU execution, resulting in unexpected performance bottlenecks.

Conclusion

On-device inference is evolving from a niche capability to a standard deployment pattern. The key takeaway is that no single framework dominates all hardware. GGUF wins on versatility, MLX on Apple-specific performance, and ONNX on cross-platform standardization. As edge hardware becomes more specialized, expect these frameworks to diverge further, with MLX and ONNX pushing deeper into NPU utilization while GGUF maintains its role as the universal fallback.