Predicting Time Horizon from Anthropic's Internal ECI

For Mythos preview and Opus 4.7

Note: While I have worked with both METR and Epoch in my capacity as a statistical consultant this post is based entirely on public information. For more information or to enquire about hiring me for a project see abstats.co.uk

Anthropic’s Internal ECI

Starting with Mythos preview Anthropic included a new ‘Anthropic ECI’ or “AECI” in their system cards. This is based on the same methodology as the Epoch Capabilities Index¹ but with additional information from their internal benchmarks.²

Unfortunately they only share the results in the form of a pretty but hard to read plot:

AECI plot taken directly from the Opus 4.7 model card

I extracted all of the values from this to obtain:

Mapping to Time Horizon

When Epoch released the ECI they noted that it was very predictive of METR’s 50% Time Horizon results.³

We can leverage this to see how AECI relates to 50% and 80% Time Horizon, to get early estimates of how Opus 4.7 and Mythos would perform:

Both show a clear relationship, especially the 50% time horizon, with a R^2 of >0.99 (for log-scale Time Horizon). We predict:

• Opus 4.7:

  • 50% Time Horizon: 18.8 Hours

  • 80% Time Horizon: 2.6 hours

• Mythos Preview

  • 50% Time Horizon: 40.3 Hours

  • 80% Time Horizon: 5.5 hours

However we shouldn’t expect to see actual 50% Time Horizon values this high from METR, as the longest task in their current TH1.1 task suite is 30 hours (and very few are over 16 hours).⁴

Thus until METR update their task suite to include more tasks, we expect they may not be able to accurately measure the 50% time horizon of the most capable models. The 80% time horizon should still fall within the measurable range however, so we will be able to compare those when they are released to these predicted values.

1. See the two previous posts on this substack for more discussion of the ECI.

2. In the version shared with Mythos Preview they also accidentally scaled 3.5 Sonnet (New) to have ECI 130 instead of the original 3.5 Sonnet release, as Epoch does. They fixed this issue in the release with the Opus 4.7 system card. If only someone could have foreseen releasing a model with the same name twice causing confusion.

3. In particular fitting the log of time horizon as a linear transformation of ECI.

4. You can view the interactive task success rate plot on this page (which I helped create) to better understand the tasks that go into calculating METR’s time horizon results.