Leaderboard

The eval

Blueprint-Bench tests spatial reasoning through converting apartment photographs into accurate 2D floor plans. Models examine ~20 interior photos and generate a floor plan showing room layouts, connections, and relative sizes.

Converting apartment photographs (left) into a 2D floor plan (right). Red dots indicate rooms, green lines show doorways.

Success requires identifying rooms, inferring spatial relationships, understanding scale, and generating structured output following strict formatting rules.

Mean similarity scores across apartments with human baselines. Humans outperform all AI systems.

Our results reveal a striking failure of current AI capabilities: most models perform at or below our random baseline (0.279), with even the best models, Gemini 2.5 Pro and GPT-5, achieve scores below the human baseline at 0.547. This significant gap suggests that while photographs are well within the training distribution of modern multimodal models, the task of spatial reconstruction—inferring room layouts, understanding connectivity, and maintaining consistent scale—requires genuine spatial intelligence that current systems lack.

Mean similarity scores across all apartments. Most perform at or near random baseline.

Many models, especially image generation systems, failed to follow the strict formatting rules required for robust scoring. This reveals challenges in instruction adherence, not just spatial reasoning.

Examples of poor instruction following: GPT-4o fails to label rooms with dots (left), NanoBanana includes forbidden details like furniture (right).

Humans worked iteratively, viewing images multiple times and refining drawings. To test if this explained AI’s poor performance, we evaluated agent systems with similar capabilities.

Claude Code agent attempting iterative refinement, making multiple attempts but still producing errors despite claiming success.

Agent-based approaches with iterative refinement capabilities showed no meaningful improvement over single-pass generation, confirming that the core limitation is spatial understanding rather than methodology.

Blueprint-Bench provides the first numerical framework for comparing spatial intelligence across LLMs, image models, and agents. As AI systems advance, spatial reasoning remains fundamental for real-world applications.

While spatial intelligence isn’t inherently dangerous, it’s a prerequisite for many advanced capabilities. Blueprint-Bench helps monitor progress toward AI systems that can truly understand physical spaces.