Hinton and Bengio urge governments to prepare for possible surge in AI research

Geoffrey Hinton and Yoshua Bengio are among more than 20 authors urging governments to prepare for a possible surge in AI progress driven by automated research. Their report calls for closer oversight of AI development and contingency plans, while acknowledging that the speed and effects of any such acceleration remain uncertain.

The report, titled ‘What if automating AI R&D triggers an intelligence explosion?’, also includes Anthropic co-founder Jack Clark and OpenAI chief scientist Jakub Pachocki, according to the Guardian. Its warning concerns what could happen if AI systems become capable of substantially advancing AI research and development themselves. The authors describe an ‘intelligence explosion’ as progress that compresses advances normally taking years into months or less.

What the authors want governments to do

The authors set out three broad priorities: make progress in AI research more visible, find ways to steer or constrain a rapid acceleration, and prepare for its wider effects. The Guardian reports that their proposed measures include transparent progress reports on AI research and development and independent auditors working inside companies. Those measures would give governments and outside evaluators more information about how quickly capabilities are changing.

Other proposals described by the Guardian include limits on how fast an AI system can improve, arrangements with data centres that could allow some research projects to be paused, and isolation of automated AI research systems. The authors also call for emergency response plans covering different ways an acceleration might unfold. These are proposals from the report, rather than measures the Guardian says governments have adopted.

Possible gains and risks

The report says faster research could lead to medical breakthroughs and other technological advances. It also warns that powerful systems could help biological and cyber threats develop faster than safeguards, reduce opportunities for people to control AI systems, and shift power between states. The authors say rapid progress could weaken checks on power within governments and companies as well as between them.

Those outcomes are uncertain, the authors acknowledge. Even if research produced a breakthrough quickly, putting it into use could take longer because of supply chains or regulatory requirements. AI might also help people develop responses to emerging risks. The report’s call for preparation rests on the possibility of a sharp acceleration, rather than a finding that its predicted consequences have begun.

What current measurements show

The Guardian quotes the authors saying productivity gains from automated AI research have not yet reached the threshold they associate with an intelligence explosion, although they think newer systems may be approaching it. The Guardian also reports their forecast that AI could fully automate research projects that take people months by 2028. That date is a forecast; the reported present-day threshold has not been crossed.

Independent measurements from METR provide context for assessing claims about AI agents, but measure something narrower. METR defines a task-completion time horizon by the length of time a human expert would need for a task that an AI agent is predicted to finish at a stated reliability. At its 50% horizon, the agent is predicted to succeed half the time. The measure concerns task difficulty, not how long the agent can work without supervision.

METR’s task set consists mainly of self-contained software engineering, machine-learning and cybersecurity work with clear success criteria. METR cautions that these results do not establish an agent’s performance across all jobs or longer, less clearly defined projects. It also says estimates of human task duration may be high compared with everyday professional work, because people in the tests have less project context.

The distinction matters for the report’s central question. Success on bounded software tasks does not establish that an AI system can conduct frontier AI research from start to finish. For now, the authors are asking governments to develop oversight and response options before a possible acceleration; when or whether one occurs, and how its benefits and risks would unfold, remains unresolved.