(1). You say: “I expect that by EOY 2026, AIs will have a 50%-reliability time horizon of years to decades on reasonably difficult easy-and-cheap-to-verify SWE tasks that don’t require much ideation (while the high reliability—for instance, 90%—time horizon will be much lower, more like hours or days than months, though this will be very sensitive to the task distribution).” If you had to guess what do you think the 50% time horizon will be at the end of 2026 as it relates to the METR graph?
(2). I noticed that in the graph with your predictions of when everything is likely the chances of an automated coder coming before ai R&D parity is higher for every year. Why is that? Shouldn’t the automated coder come after ai R&D parity?
Easy versus hard verification seems like a fuzzy boundary that I would expect keeps getting pushed out by making verification cheaper, model capability increases, better harnesses. I think this matters for superexponentiality because turning hard verification into easy verification is part of a SWEs job, so it seems likely to me that hard to verify tasks could become easy to verify with enough engineering effort.
great stuff! been thinking along many similar lines
Great post! I did have some questions.
(1). You say: “I expect that by EOY 2026, AIs will have a 50%-reliability time horizon of years to decades on reasonably difficult easy-and-cheap-to-verify SWE tasks that don’t require much ideation (while the high reliability—for instance, 90%—time horizon will be much lower, more like hours or days than months, though this will be very sensitive to the task distribution).” If you had to guess what do you think the 50% time horizon will be at the end of 2026 as it relates to the METR graph?
(2). I noticed that in the graph with your predictions of when everything is likely the chances of an automated coder coming before ai R&D parity is higher for every year. Why is that? Shouldn’t the automated coder come after ai R&D parity?
Easy versus hard verification seems like a fuzzy boundary that I would expect keeps getting pushed out by making verification cheaper, model capability increases, better harnesses. I think this matters for superexponentiality because turning hard verification into easy verification is part of a SWEs job, so it seems likely to me that hard to verify tasks could become easy to verify with enough engineering effort.