Blueprint-Bench 2

Blueprint-Bench 2 tests spatial reasoning by asking AI agents to convert apartment photographs into accurate 2D floor plans. Each agent processes 50 apartments sequentially, examining ~20 interior photos per apartment and generating a floor plan showing room layouts, connections, and relative sizes. Agents use a persistent notepad to carry insights between apartments, enabling cross-apartment learning and iterative strategy refinement.

All scores are normalized so that the random baseline maps to 0 and a perfect score maps to 1. Error bars represent standard error.

Leaderboard

	Model	Score
T1	Gemini 4 Carbon	0.586
T1	Human*	0.586
3	Gemini 4 Argon	0.544
4	Claude Opus 5.5	0.512
5	GPT-6 Astra	0.497
6	GPT-6.1 Terra	0.421
7	Claude Fable 5.1	0.419
8	Claude Sonnet 5.5	0.393
T9	Claude Fable 5	0.386
T9	Gemini 3.8 Flash	0.386
11	GPT-6 Sol	0.369
12	GPT-5.5	0.362
T13	Gemini 3.5 Flash	0.336
T13	GPT-5.6 Sol	0.336
15	Grok 4.6	0.332
16	Claude Haiku 5.5	0.330
17	Grok 4.7	0.325
T18	Gemini 3.6 Flash	0.312
T18	GPT-6 Luna	0.312
20	GPT-5.6 Terra	0.308
21	Claude Opus 5	0.304
22	Kimi K3	0.295
23	Grok 4.5	0.273
24	GPT-5.4	0.271
25	Gemini 3.1 Pro	0.265
26	Claude Sonnet 5	0.249
27	Claude Opus 4.7	0.245
28	GPT-5.6 Luna	0.226
29	Claude Opus 4.8	0.145
30	Claude Sonnet 4.6	0.067
31	Kimi K2.6	0.039
T32	Gemini 3 Flash	0.000**
T32	Grok 4.3	0.000**
T32	Gemini Robotics-ER 1.6	0.000**
T32	Claude Haiku 4.5	0.000**
T32	Grok 4.20 Reasoning	0.000**

**Score at or below the random baseline

*Human baseline tested on subset of 12 apartments only

SOTA frontier models are labeled and a trend line is fitted through them, with a projection into the near future.

The dashed blue line is the human baseline (0.59). Gemini 4 Carbon now ties it.

The eval

Blueprint-Bench 2 tests spatial reasoning through converting apartment photographs into accurate 2D floor plans. Models examine ~20 interior photos and generate a floor plan showing room layouts, connections, and relative sizes.

Converting apartment photographs (left) into a 2D floor plan (right). Red dots indicate rooms, green lines show doorways.

Success requires identifying rooms, inferring spatial relationships, understanding scale, and generating structured output following strict formatting rules.

Scoring

Plans are scored by comparing their connectivity graph against ground truth. The composite score weights six sub-metrics: Jaccard similarity (50%) measures overlap in room-to-room connections, degree similarity (20%) compares how many doors each room has, density similarity (10%) checks overall graph connectivity, room count (10%), door count (5%), and orientation (5%). All scores are matched under D4 symmetry to be rotation and reflection invariant. Scores are then normalized so that the random baseline maps to 0 and a perfect score maps to 1.

Agent notepad system

Each agent processes 50 apartments sequentially and has access to a persistent notepad. This notepad carries across apartments, letting agents record strategies, common patterns, and lessons learned. The best models use this to build structured knowledge about typical apartment layouts, improving their approach over time.

Key findings

Blueprint-Bench 2 was released in May 2026. The top three models (GPT 5.5, Gemini 3.1 Pro, and Claude Opus 4.7) significantly outperform the rest, with tight variance indicating consistent spatial reasoning. The key discriminator is Jaccard similarity (room-to-room connectivity). All models achieve ~90% on room count, but lower-performing models struggle to correctly infer which rooms connect to which.

Gemini Robotics-ER 1.6 underperforms expectations. Despite being designed for spatial and embodied reasoning, it scores below Gemini 3 Flash. Its spatial specialization does not translate to improved floor plan generation.

Sparks of spatial reasoning

In the original Blueprint-Bench, model outputs were essentially noise. In Blueprint-Bench 2, we see the first signs of genuine spatial reasoning from 2D photographs.

Reversing camera direction using landmarks - Gemini 3.1 Pro uses a washer/dryer visible in two photos to figure out which direction the camera is facing:

assistant · Gemini 3.1 Pro

If the washer/dryer is on the right, it means we are looking from the front door towards the living room. (...) And if you face away from the door, the washer/dryer is on the right, the bedroom is straight ahead, the living room is on the left. Bingo!

Inferring a through-room from multiple doorways - GPT 5.5 notices doors in two photos of the same bedroom lead to different rooms, deducing it functions as a connecting passage:

assistant · GPT 5.5

In image six, I notice a clear door between the bedroom and living area, indicating direct connection. (...) In image seven, I see a closed door on the left wall, possibly a closet, and the central open door connects to the hall (...) This suggests the bedroom might function as a through-room, connecting to both the living area and the hall.

@misc{andonlabs2026blueprintbench2,
  title={Blueprint-Bench 2},
  author={Andon Labs},
  year={2026},
  url={https://andonlabs.com/evals/blueprint-bench-2}
}