EvalCore

EvalCore is a lightweight CLI tool that helps you catch when your AI app starts behaving worse — before your users notice. You define evaluation cases in a simple YAML file plus a JSONL dataset, run it once against your real model to record every request and response in a local SQLite cache, then replay that recording on every future code or prompt change. The replay is offline, deterministic, and costs nothing in CI, so your team gets fast, reliable feedback without flaky tests or API bills.

What makes EvalCore stand out is how minimal it is. There's no SDK to integrate, no test framework to learn — just a single binary that speaks HTTP or shell to your app, whatever language it's written in. It works with OpenAI, vLLM, Ollama, any OpenAI-compatible gateway, or your own REST APIs. You can stack scorers (like checking for a keyword or using a judge rubric) to define what "good" means for each case. The record/replay model means the first live run captures cost and latency, and every subsequent CI run is instant and free.

This tool is built for developers shipping AI features — whether you're a solo dev iterating on prompts or a team managing multiple model versions. It's especially useful when you swap models, tweak prompts, or update dependencies and want to know immediately if anything broke. Because it runs fully offline in CI with no API keys, it fits naturally into existing PR workflows and gates on exit codes. If you've ever shipped a prompt change that silently made your chatbot dumber, EvalCore is the safety net you didn't know you needed.