PX-bench

PX-bench is a benchmark designed to measure how well AI coding agents handle real-world product experience decisions — not just whether the code runs. Most coding benchmarks focus on correctness alone, but PX-bench goes further: it evaluates agents on intent fidelity, product fit, visual craft, convention adherence, pathway completeness, content and language, resilience, and accessibility. Each category captures a kind of judgment a senior product designer would make when adding a feature to an existing app.

The key idea is that agents don't build from scratch. Instead, they add a feature to a held-out, multi-screen host app with its own established conventions. This forces the agent to reuse components, follow design tokens, and place features in the right context — things that don't show up in a typical "write a function" test. The benchmark maps likely failure modes in advance and scores the agent's choices against a known-good outcome defined by expert product designers. The rubrics are designed to be quasi-objective: only included when senior designers agree on the right answer.

PX-bench is built on the Inspect AI framework from the UK AI Safety Institute, so any scenario can be independently rerun. It's most useful for teams building or fine-tuning coding agents who want to catch regressions, reduce cost without hurting quality, and understand where their agent's product sense breaks down. If you're shipping an agent that writes code for real users, PX-bench gives you a scorecard that goes beyond "does it compile?"