‘No High-Risk Area Is as Poorly Regulated as AI’

Several viewpoints are circulating in public discourse: Some say AI is highly dangerous and there is a realistic danger that it will lead to the extinction of humanity. Others regard such statements as calculated PR. How serious do you think these dangers are?

For more than ten years, leading scientists have been warning us that there may be an – initially still unclear or very low – probability of uncontrolled self-improvement in highly powerful AI models. These are not people who are against scientific progress or against economic development, and I consider their arguments to be absolutely convincing. In 2026, we seem to already be in the moderate-risk range, perhaps even in the range of a concrete danger, not only that of a minimal or unclear risk. While the US president did speak of a ‘hoax’, he emphasized in the same breath that the USA must not give up its lead against China. It seems to me that this is more likely the reason why these risks are not being taken into account in the USA and are not being reduced through regulation. Yet it is the responsibility of governments and states to mitigate high risks and protect the people from them. Furthermore, there is, in my mind, a duty to protect rooted in human rights, a legal obligation of states based on applicable international human rights treaties. We also see that states mitigate risks for the people in other areas: the approval of drugs is tightly regulated; biotechnology is regulated; civil use of nuclear energy was and still is tightly regulated – also through liability provisions. It is, in my mind, negligent and wrong to leave these questions for companies to solve: It is the responsibility of states and legislatures to formulate rules to mitigate special risks, and also to decide which AI risks we as societies are willing to take for which uses of AI. It cannot be that we need to be on the brink of a catastrophe to wake up, as in the case of the bank crisis. The current debate is therefore good and important.

What kind of regulation and control do we need to prevent these dangerous developments?

The best solution would certainly be an international treaty, so that all relevant states can be bound in equal measure. What we should do here is establish a framework that provides for an independent commission of experts to assess the risks and benefits of frontier AI models before they are introduced on the market. The relevant states and their companies should be required to make these high-risk models available for evaluation and to report them to this commission of experts. According to this model, which I developed together with Thorsten Schmidt and published in 2022, it should also be mandatory to establish a fund into which each company must deposit so-called regulatory capital before being allowed to introduce a high-risk model on the market. The purpose of this fund would be to cover losses that could occur over the following five years – as a means of protecting the public but also as an incentive for the company to reduce these risks. If no damages occur during this period – which we all hope will be the case – the capital must and may be repaid to the AI company.

Is it still possible to negotiate international treaties at all today?

There have been successful treaty negotiations in recent years, for example on the use of genetic resources on the high seas, the so-called BBNJ Agreement. Important treaties were concluded even during the Cold War, such as on the peaceful use of nuclear energy or in space law. And again, we do not yet have sufficient international standards to lower high AI-based risks. On the contrary: There is no high-risk technology sector that is as poorly regulated under international law as AI. We find so-called soft law standards, but they are not binding in a strict sense and are not aimed at high risks and their mitigation. A human rights-based treaty has been negotiated within the context of the Council of Europe, but it is not yet in force and is also too general to mitigate this kind of AI risks.

‘If there are going to be killer robots, I would rather they be American killer robots, rather than Chinese killer robots’ – that’s what the US Republican Senator Ted Cruz recently said in a Politico interview on the topic of AI. In his estimation, it is not possible to slow down the pace of AI development on account of international competition…

What I would ask him in response is whether he would also be comfortable with killer robots controlled by AI agents that cannot be stopped or controlled by the USA or China… After all, that is the particular threat posed by high-risk agentic AI. In addition, if there is a new international treaty, all relevant states should be bound by it. If there are companies in the USA developing high-risk products, however, there is, in my view, already a human rights obligation to avert and mitigate risks to people’s lives and physical well-being – by which I mean legally binding human rights based on treaties ratified by the USA. Moreover, weapon development is a very specific area that does not encompass the entire extent of high-risk AI but only a fraction of it. If necessary, this area can be left aside, also because very specific talks have been taking place in Geneva for some time now, which may lead to the negotiation of a treaty.

There have already been initial attempts to regulate AI, such as the EU AI Act. How do you rate the existing AI laws in Germany and the EU?

The EU AI Act follows the right goals: protection of human rights and regulation that is appropriate to the risks. In my view, however, this regulatory framework is unfortunately much too complicated, and yet it is still incapable of mitigating the relevant dangers of high-risk agentic AI and frontier models, either because they are not covered by the regulations or because the standards lack sufficient rigour. Unlike in the area of drug approval, for example, there is no real approval process and no specific supplementary liability regime. By contrast, we also see very technical international regulations in specific areas, such as the standards for autonomous vehicles: In this area, we have German law, clear registration procedures, liability rules, and data transmission rules in the event of accidents, and there even treaties under international law by the United Nations Economic Commission for Europe (UNECE). Hence, regulation can be implemented quickly enough and work effectively if all stakeholders are willing.

Do these laws perhaps prevent European AI from keeping up with the international competition?

I consider that to be an unsubstantiated claim: German or European pharmaceutical manufacturers are not less successful because there is an approval procedure for drugs. In addition, we did not have any AI regulation for years, until 2024, and yet there was no company here that could truly compete with OpenAI or the Chinese AI companies. The key point here, it seems to me, is how money can be invested in new ideas and sectors and what money is available for that purpose. Moreover, we are at the international forefront in other sectors that have been subject to clear and swift regulations, such as autonomous vehicles. Regulations that are clear, proportionate, and not overly complex help us all, also because industry, research, and development can adapt to them.