OpenAI Learns to Limbo
You’ve probably heard of Goodhart’s Law—the idea that, once a measure becomes a target it ceases to be a good measure. Well guess what, Mythos-level benchmark performance just became the target to avoid. And would you look at that? OpenAI’s latest frontier model, GPT 5.6 Sol, landed right under the target.
Did it work? At the risk of spoiling the ending—not perfectly. If you caught Theo’s newsletter yesterday, you’ll know that GPT 5.6 has been released but, at the request of the US Government, only to a list of partners pre-approved by the government itself. At the same time we heard that Anthropic has been given the go ahead, also by the government, to reopen access to Mythos 5 (the Mythos model checkpoint that followed Preview and excludes the safeguards that made Fable 5 rather unwieldy) for a list of over 100 “trusted partners”. Reporting suggests this list might have substantial overlap with Project Glasswing, but it’s not yet clear whether there are new additions.
But counterfactuals are hard. It seems very possible to me that a flashier release showing capability advancements over Mythos, or even just emphasising Mythos-level risks in the house style of Anthropic—artful anecdotes about sandwiches in parks disturbed by emails from advanced models, looming dark clouds on the horizon—could have provoked more drastic sanctions from the USG.
What did they do instead? If you look at the chart below, you will see several lines representing the performance of OpenAI’s new models on ExploitBench, an offensive cybersecurity benchmark. Cybersecurity concerns, we should remember, motivated the decision by the USG to place an export directive on Fable 5. The y-axis shows performance on the benchmark for a given number of output tokens, shown on the x-axis.
Tracking the models’ rightward over the x-axis demonstrates how their performance improves with additional inference, or test-time compute. Older models demonstrate improved performance with additional output tokens, but also diminishing returns that kick in at relatively low levels of test-time compute. GPT-5.5, for example, starts showing a flatter inference response shortly after 100k output tokens.
Take a look at the topmost line, which represents the performance of OpenAI’s frontier, highest-capability model, GPT-5.6 Sol. It looks to me like Sol is quite far from diminishing returns here, and a larger token budget would have seen it match or exceed the line on the chart representing Mythos 5’s performance. This is what 0xhorror is suggesting in their tweet above—I saw someone else refer to it as a ‘bench-min’ strategy. Make your models look smolbean and unthreatening, and perhaps you’ll be able to release it to users.
In fact, OpenAI researcher Noam Brown has raised this himself. In a longer post about test-time compute and advanced capabilities, Noam says:
Before a frontier model is released, labs typically evaluate cyber, bio, and other misuse risks. If a model crosses a capability threshold, then release may be delayed until mitigations are in place. But if capability is a function of inference compute, then at what inference budget should safety evaluations be run?
Perhaps if you wanted to avoid crossing a capability threshold you could work backwards from threshold to inference budget. Curious. Much to consider.
Of course, I don’t know whether this is true or not. There may well be other reasons for the results, some of them less suggestive of Altman 5D chess. But it does bring to mind a broader difference between the approach taken by the two competing frontier labs.
If you have spoken to an OpenAI researcher since the Mythos story broke, you will have heard Noam’s words from their mouths. If you ask, for example, why Anthropic has a Mythos-tier model and OpenAI doesn’t, they might say something like: Mythos is a very good model—it has ‘big model smell’, the feeling that you are talking to a very bright human. But if what you care about is autonomous, agentic performance of tasks, ‘big model smell’ can be misleading. Comparing model capabilities without a token budget is meaningless—if you want Mythos level performance from 5.5 just run it for longer.
My amateur speculation is that this reflects, in part, diverging research directions at the two labs. Simplistically, Anthropic has focused on pre-training large and highly capable base models. OpenAI has placed relatively more importance on techniques that allow smaller, less expensive models to complete tasks over longer time horizons and take better advantage of test-time compute. I don’t think that benchmin-ing was the goal of this research direction, but, in the present climate, it seems like a useful side effect.
I expect we will see more of this over future OpenAI model releases—test-compute scaling now looks like a marginally safer method of progress than training highly capable big models. If we start seeing signs from this Administration that they’re buying this—weaker restrictions placed on Sol than on Fable 5, despite similar capabilities on frontier cyber tasks, for example—OpenAI may eschew large model releases entirely, or keep them strictly for internal research enhancement. Something to watch.
One promising sign: OpenAI has made good use of this feature of their latest models in their public comms around the model release.
“We have made a good model. It is nice and efficient. I would definitely not describe it as too dangerous to release. It does seem to scale beautifully with additional inference however. Oh and, just by the way, it’ll be available at seven times the speed of GPT-5.5xHigh.”
Very smart, and good to see. Time will tell whether, as a strategy, it might be a little too smart. It may well be a “if those kids could read, they’d be very upset” moment when it comes to the Trump Administration. Time, as always, will tell.