Omlx AI

Omlx AI is a local LLM inference server built specifically for Apple Silicon Macs. It uses the MLX framework to run large language models directly on your machine, with a focus on low latency and high throughput for coding agents like Claude Code, OpenClaw, and Cursor. Instead of waiting 30–90 seconds for a response, Omlx AI promises time-to-first-token under 5 seconds on the second turn, thanks to persistent SSD caching.

The key innovation is paged SSD KV caching. Traditional local inference tools like Ollama and LM Studio store the key-value cache in memory, which gets invalidated when the context shifts mid-session. Omlx AI saves every cache block to disk, so when a coding agent loops back to a previous context, the cached data is restored from SSD in milliseconds. This dramatically reduces recomputation. The tool also supports continuous batching, handling multiple concurrent requests with up to 4.14× speedup at 8× concurrency. It runs as a native macOS menu bar app (not Electron) with a web dashboard for managing models, monitoring performance, and generating API config commands.

Omlx AI is compatible with OpenAI and Anthropic API formats, so it works as a drop-in backend for any client that supports those standards. It can serve multiple model types simultaneously — LLMs, VLMs, embedding models, and rerankers — and includes tool calling support for JSON, Qwen, Gemma, GLM, MiniMax, and MCP. The target audience is developers and AI researchers who want to run large models locally on a Mac without the long wait times typical of memory-only caching. It is especially useful for anyone using coding agents that require frequent context switching, such as software engineers experimenting with autonomous coding, or researchers testing reasoning models like DeepSeek and Qwen. Installation is straightforward: download the DMG or install from source, and it automatically reads your existing Hugging Face cache, so no model re-download is needed.