Roboflow benchmarks OpenAI GPT-5.6 vision models

Roboflow has published a comprehensive benchmark evaluating OpenAI’s new GPT-5.6 vision model lineup—Sol, Terra, and Luna—against competitors including Google’s Gemini 3.5 Flash and Anthropic’s Claude Fable 5. The analysis identifies GPT-5.6 Sol as OpenAI’s most capable vision model to date, demonstrating significant gains in object detection and counting accuracy compared to the previous GPT-5.5 generation. Despite these performance improvements, the GPT-5.6 models incur higher costs and latency, with Sol averaging 2.5 cents per image.

In object detection, the GPT-5.6 Sol model achieved a mean Average Precision (mAP@50) of 46.2, a substantial increase from GPT-5.5’s score of 13.8. Terra and Luna scored 44.7 and 43.3 respectively, moving object detection from a major weakness to a practical capability for the OpenAI suite. The models also showed improved document layout detection, handling titles, paragraphs, tables, and signatures effectively. However, performance on dense scenes with many similar objects remained challenging, with the risk of missed objects or coordinate errors increasing as object count grew.

Counting accuracy improved across the full lineup, with Sol scoring 73.0%, Terra 67.6%, and Luna 66.2%, compared to GPT-5.5’s 64.9%. Sol demonstrated the ability to count heavily overlapping metal brackets and bullet holes within specific scoring zones, though it struggled with blister packs due to reflections and small visual differences. OCR performance remained largely static, with Sol achieving a 90.7% mean similarity score, slightly behind GPT-5.5’s 91.2%.

The GPT-5.6 models exhibit higher token usage, latency, and cost compared to previous versions. Sol averaged 10 seconds per image, while Terra and Luna reduced latency to 6 and 5 seconds respectively. Luna offers the strongest latency-quality balance in the lineup, with speed close to Gemini 3.5 Flash while still outperforming GPT-5.5 on detection and counting. At 0.8 cents per image, Gemini 3.5 Flash is much cheaper than Sol while still leading in specific benchmarks, making it a strong option for data-intensive workloads where cost scales across large image batches.

OpenAI confirmed that Sol becomes less stable on images around 2,000 by 2,000 pixels or larger, particularly at lower reasoning efforts. Using absolute XYXY coordinates is recommended for GPT-5.6 detection, whereas Gemini 3.5 Flash performs best with YXYX coordinates normalized to a 0–1000 range. The benchmark highlights that while OpenAI is now taking vision much more seriously, clear limits remain regarding cost, latency, and stability in certain conditions.