Fable 5 outperforms GPT-5.6 Sol in NP-hard optimization benchmark; goal mode shows mixed results
While Fable 5 demonstrated superior consistency and achieved the highest individual score, the native goal feature failed to improve average performance across trials, occasionally amplifying poor decisions.
Charles Azam has released new benchmarking data comparing Claude Fable 5 and GPT-5.6 Sol on an unpublished NP-hard optimization problem known as KIRO. The test, which evaluated the impact of native goal mode features, found that Fable 5 delivered superior consistency and produced the best overall solution. The results suggest that while specific model architectures offer distinct advantages, persistence features do not universally enhance performance on complex tasks.
The KIRO problem involves designing fiber-network loops for cities including Grenoble, Nice, and Paris. The objective is to connect distribution points and terminals using loops and short chains while respecting structural constraints to minimise total cable length. Azam conducted three matched runs for each flagship model, alongside a broader sweep of other models, to assess how the goal feature influenced the search path and control loop.
Fable 5 exhibited remarkable stability, with its plain configuration staying within a 319-point range. In contrast, GPT-5.6 Sol showed higher volatility, with a plain configuration spanning 1,958 points. Fable 5’s plain mean score beat GPT-5.6 Sol’s by 1,875 points, and its goal mean beat Sol’s by 1,984 points. The highest individual score achieved was 31,934 by Fable 5 using goal mode.
Despite Fable 5’s dominance, the goal feature did not significantly improve average performance for either model. Although it won four of the six individual trials, it occasionally amplified poor decisions, resulting in worse mean scores for both models compared to their non-goal counterparts. This indicates that while the feature can help sustain good decisions, it can also extend the duration of suboptimal strategies.
The technical implementation of the goal feature differs between the models. Claude Code uses a session-scoped Stop hook with an evaluator model that judges the transcript, while Codex treats the goal as persisted thread state. Azam noted that on normal coding tasks, progress is often legible, but on optimization problems, extra time can amplify either a good decision or a bad one.
The benchmark environment exposed eight CPUs despite task metadata declaring one, which may have favoured Fable 5’s parallel processing capabilities. All scored outputs from Fable 5 and GPT-5.6 Sol were verified as valid, partly due to wrapper requirements for early checkpoints and final verification. The full data, including code, prompts, and trajectory notes, is available in CLIArena.
The findings highlight a critical distinction in model evaluation: a persistence feature can win most individual trials while making observed average performance worse. On hard optimization problems, the quality of the loop matters less than the quality of what the loop keeps doing. This suggests that investors and developers should look beyond simple win rates when assessing the utility of new model features.
