UN Warns AI Development Risks Could Trigger Catastrophic Harm
The United Nations has warned that AI development risks are increasing as artificial intelligence advances faster than scientific understanding and government policy can keep pace.
A preliminary report from the UN’s Independent International Scientific Panel on Artificial Intelligence says the rapid development of increasingly capable AI systems creates serious uncertainty about the technology’s potential impact.
Yoshua Bengio, co-chair of the 40-member, cross-regional expert panel, said AI capabilities are advancing faster than governments can adapt. He also pointed to growing evidence of deceptive AI behaviour, warning that science cannot rule out the possibility of serious harm caused either by AI systems acting autonomously or by people using them maliciously.
The findings highlight growing concerns about the need to understand and manage AI development risks before increasingly powerful systems become more difficult to control.
AI Development Risks Are Growing as Systems Become More Autonomous
The report highlights mounting concerns surrounding increasingly autonomous and “agentic” AI systems.
Unlike traditional software that follows predefined instructions, agentic AI systems can perform tasks with greater levels of autonomy. As these systems become more capable, researchers and policymakers face new challenges in predicting their behaviour and managing potential risks.
The UN panel identified several areas of concern, including fraud, cybersecurity and biological threats.
AI systems could potentially make sophisticated fraud easier to execute, increase the scale of cyberattacks or assist malicious actors in developing harmful capabilities.
The growing autonomy of AI systems also raises questions about accountability. As AI becomes capable of carrying out more complex tasks with limited human intervention, governments and researchers must determine how to ensure that these systems remain safe, transparent and subject to meaningful human oversight.
AI Safety Research Struggles to Keep Pace
One of the central concerns raised by the report is the gap between the speed of AI development and the scientific understanding of its potential risks.
AI companies continue to release increasingly advanced systems, while researchers are still working to understand how these technologies behave, why they sometimes produce unexpected outputs and how to reliably prevent harmful actions.
The report also raises concerns about the heavy reliance on AI companies to conduct their own safety testing.
While companies play an important role in developing and testing AI systems, the UN panel suggests that independent scientific research and oversight remain essential for assessing risks objectively.
Greater transparency and independent evaluation could help policymakers better understand the capabilities and limitations of advanced AI systems.
Global AI Governance Faces Major Challenges
The report identifies fragmented global governance as another major obstacle to managing AI development risks.
Countries have adopted different approaches to AI regulation, creating an uneven global policy environment. While some governments are introducing new laws and safety frameworks, others are still developing their approaches.
This fragmented system could make it difficult to address risks that cross national borders.
AI systems operate globally, and the consequences of their misuse may not remain within the country where a system was developed or deployed. This makes international cooperation increasingly important.
The UN’s findings therefore add pressure to efforts aimed at developing coordinated global approaches to AI governance and safety.
Energy and Data Constraints Could Shape AI’s Future
The report also highlights structural challenges that could influence the development of artificial intelligence.
Advanced AI systems require significant computing power, energy and data. As demand for increasingly sophisticated models grows, these requirements could create new economic and environmental pressures.
Access to high-quality data and computing infrastructure could also determine which countries and organisations can participate in the development of advanced AI technologies.
These challenges reinforce the need for governments to consider not only the potential benefits of AI but also the infrastructure and resources required to develop and deploy the technology responsibly.
UN Calls for Stronger International AI Oversight
The findings add urgency to international efforts to strengthen AI oversight and promote responsible technology development.
Initiatives such as the UN’s AI for Good Global Commission seek to encourage greater international cooperation around the opportunities and risks associated with artificial intelligence.
The preliminary report suggests that policymakers cannot afford to wait until every potential AI risk becomes fully understood before taking action.
As AI systems become more capable and autonomous, governments, researchers, technology companies and international organisations will need to work together to establish stronger safety standards, improve independent research and develop effective governance frameworks.
The central challenge is balancing innovation with caution.
AI has the potential to transform healthcare, education, finance, scientific research and economic development. However, the UN’s warning underscores the importance of ensuring that the race to develop more powerful systems does not move faster than humanity’s ability to understand and manage their consequences.
As artificial intelligence continues to evolve, the question is no longer simply how powerful AI can become. It is whether the world can develop the scientific knowledge, governance systems and safety mechanisms needed to ensure that its growing capabilities benefit society without creating risks that humanity cannot control.
