Part C

Assessment of uncertainty in learned models, risk management, ethical and legal basis and implications

WP6: Estimating uncertainty in learned models

WP6 addresses the challenge of estimating quantifying in learned models for sequential decision processes such as reinforcement learning (RL) and Bayesian optimization (BO). To this end, this work package develops and benchmarks advanced methods for uncertainty quantification in deep learning, enabling BO and RL algorithms (including robots) to make safer and more informed decisions during sequential learning in complex real-world environments. By creating reliable and robust uncertainty estimates, WP6 ensures more effective exploration and stable learning leading to improved data efficiency and robustness in these algorithms. WP6 collaborates with other WPs to enhance the safety, scalability, and efficiency of autonomous systems, supporting lifelong learning and improvement in robotic applications.

Recent Highlights from WP6

WP6 (Uncertainty estimation) reported progress on benchmarking uncertainty in model-based RL with rich visualizations. A new partial observability benchmark analyzed agents’ belief estimation and out-of-distribution generalization. Additional studies involve training RL agents on one-shot world models and collaborations linking WP1 and WP7.

Visualizing what trained agents learn in different environments

DeepMind Control Suite Environment: Walker walk

True rollouts ➔

Imaginary rollout samples ➔

Figure 1: *Capturing model uncertainty: Samples visualizing the distribution learnt over future states for a trained model of the Walker Walk environment from the DeepMind Control Suite.*

True rollouts ➔

Imaginary rollout➔

Differences➔

Figure 2 *shows the difference between one of the samples from the trained model and the ground truth trajectories.*

We see 6 different ground truth trajectories in the upper rows, while we see a reconstruction of it using a learned model based on pixel inputs trained using the DreamerV3 agent. DreamerV3 uses a Convolutional Neural Network (CNN)-based encoder to first encode the input observations into a lower-dimensional latent space that models a categorical distribution (Fig. 1.). Samples from this distribution are plotted in the 3 rows below (Fig. 2.), each one independently sampled from the learnt categorical distribution. DreamerV3 uses a CNN-based decoder to get from the latent space back to the pixel-based observation space such that the samples can be visualized here.

Figure 3: *Comparison of learnt belief vs oracle belief in a partially observable environment.*

In the leftmost plot, we visualize a particular timestep in the DreamerV3 agent’s trajectory in the MordorHike toy environment. An agent’s ground truth trajectory is visualized as a red line with orange dots representing the belief of the agent’s position using a particle filter (that has access to a partially observable version of the environment’s ground truth dynamics and acts as an oracle in partially observable settings).
In the center plot, adiscretized grid is used to marginalize over the particles of the particle filter giving us the heatmap, illustrating the probability mass over the X and Y coordinates. This acts as an oracle belief to which we can compare learnt models such as DreamerV3.
In the rightmost plot for comparison, a linear probe is used to predict this oracle belief using DreamerV3’s learnt belief representation and we see that while DreamerV3 does not capture the oracle belief distribution perfectly, it does capture a bi-modal distribution.

WP7: Risk management and safety in human-robot interaction

WP7 focuses on ensuring safety and risk management in human-robot interaction. It develops adaptive frameworks to identify, monitor, and mitigate risks such as hazardous behavior, accidents, or manipulation. By incorporating expert knowledge, uncertainty estimation, and risk assessment tools, WP7 aims to create reliable, transparent procedures that foster trustworthiness. This work supports the safe deployment of AI-driven robots through governance frameworks, permit procedures, and risk-sensitive training, collaborating closely with other WPs to establish secure and trustworthy human-robot collaboration.

Recent Highlights from WP7

WP7 advances risk assessment through two complementary research streams. First, it develops a novel Bayesian estimation framework incorporating expert opinions as set-valued observations, alongside AI regulation and financial risk models based on Stackelberg equilibria. Future extensions will introduce partial observability, enabling sophisticated expert opinion modeling in strategic contexts.

Second, statistical risk analysis on WP2 robotics datasets reveals a bimodal cost structure, leading to refined hierarchical Bayesian models that capture task-specific variation. Planned extensions will incorporate random cost maps with interval-distributed costs, moving beyond point-mass approximations to better represent real-world uncertainty.

Figure 4: *The figure above illustrates the posterior predictive distribution across different cost categories, showing the probability distribution of potential outcomes based on our Bayesian analysis.*

Figure 5 presents the overall cost distribution alongside two critical risk measures: Value at Risk (VaR) and Expected Shortfall (ES). These risk measures provide essential guidance on **capital allocation**. Specifically, they indicate the amount of money that should be set aside to adequately cover the most likely losses. In risk management, VaR represents the maximum expected loss at a given confidence level, while Expected Shortfall estimates the average loss in worst-case scenarios beyond the VaR threshold.

Key Observation: Data Scarcity Challenge

A critical insight emerges from our analysis: both risk measures converge at $100 (worst-case scenario). This convergence immediately reveals a fundamental challenge in our current dataset—insufficient data makes it difficult to establish a clear and reliable picture of the loss distribution.

Our Solution: This is precisely where our work on expert opinions and synthetic data generation becomes invaluable. By incorporating expert knowledge and generating additional representative data points, we can enhance the robustness of our risk estimates and provide more reliable guidance for financial planning.

WP8: Governing responsible AI-based robot behavior

WP 8 aims to propose a governance scheme, including basic ethical and legal requirements and conditions for implementing a responsible research and innovation approach to the envisioned AI-based robotic systems, based on existing regulations, especially in the context of the EU General Data Protection Regulation (GDPR) and the EU AI ACT.

To map important ethical requirements for human-centred robot design, we use an interdisciplinary and participatory research approach. To this end, a group of different stakeholders with technical, ethical and legal expertise co-designed and role-played robot behaviour and human-robot interaction in various scenarios. In a subsequent focus group, participants discussed their experiences in interacting with the robot designs. In the discussion, ethical challenges, such as bias and fairness of current and future robot designs, e.g., involving large language model modules for speech interaction, were discussed.

In the legal part of this work package, we are conducting ethical and legal analysis of design requirements of robots, by considering different perspectives. One approach is to look at the regulation of humanoid social robots from a public international law perspective. How can the anticipatory governance of social robots be achieved by combining key elements of regulatory regimes from human rights law, environmental law, and their interactions? Furthermore, we are interested in how the development and use of robots can be aligned with human rights. This is particularly linked to the question of transparency in AI applications as the alignment with human rights cannot be ensured without mechanisms that make the functioning and decision-making processes of AI systems understandable and accountable. We therefore argue that a human-rights-based approach to robotics requires not only technical safeguards but also institutional frameworks that ensure transparency, accountability, and public participation in the governance of AI systems. Besides, robots can be controlled using neurotechnologies, which raises further human rights challenges, for example with regard to mental privacy and freedom of thought. In light of these new challenges for the human rights of the „mind“, we propose a new interpretation of the human right to freedom of thought. Another central part of our work is analysing existing technology regulation of the European Union, in particular the new Artificial Intelligence Act.

Recent Highlights from WP8

In the first part of WP8, Participatory Co-Design and Ethics by Design, the research advanced the conceptual framework of co-design and co-creation with emphasis on designing with and by people. They identified the challenge of equitable and fair co-design and co-research. One proposed solution was to create novel hybrid “third spaces” to create room for connecting expert and social user perspectives.

Applying Third Space Theory to Human-Robot Interaction (HRI)

Focus: Deconstructing knowledge hierarchies in the context of human-robot interaction.

One key aspect of this approach is the negotiation of meaning instead of valuing concepts of one expertise over the other.

In the second part of WP8, Ethical and Legal Analysis of Design Requirements, the researchers analyzed the Council of Europe AI Convention, comparing it to the EU AI Act, with a focus on various articles related to human dignity and autonomy, transparency and explainability, and risk and impact management. The Convention, complementary to the EU AI Act, reinforces the human-rights-based governance of AI. Potential collaboration with WP9 on the interpretability and explainability of AI systems was identified.

The approach is to look at the regulation of humanoid social robots from a public international law perspective. How can the anticipatory governance of social robots be achieved by combining key elements of regulatory regimes from human rights law, environmental law, and their interactions? Furthermore, we are interested in how the development and use of robots can be aligned with human rights. One paper discussing fairness and bias in robot learning and AI discrimination has been published: Londoño et al., Fairness and Bias in Robot Learning Proceedings of the IEEE, May 2024.

This is particularly linked to the question of transparency in AI applications as the alignment with human rights cannot be ensured without mechanisms that make the functioning and decision-making processes of AI systems understandable and accountable. Transparency provides the basis for assessing whether robots respect principles such as dignity, autonomy, privacy, and equality. Without this, potential biases, discriminatory outcomes, or violations of fundamental freedoms remain hidden, undermining public trust and ethical legitimacy. We therefore argue that a human-rights-based approach to robotics requires not only technical safeguards but also institutional frameworks that ensure transparency, accountability, and public participation in the governance of AI systems: Feuerstack, D., Menschenrechtliche Vorgaben an die Transparenz KI-basierter Entscheidungen und deren Berücksichtigung in bestehenden Regulierungsansätzen, in M. Löwisch, Th. Würtenberger, M.E. Geis, D. Heckmann (Hrsg.), Künstliche Intelligenz in Forschung, Lehre und Hochschule, (March 2025), 155.

Besides, robots can be controlled using neurotechnologies, which raises further human rights challenges, for example with regard to mental privacy and freedom of thought. In light of these new challenges for the human rights of the „mind“, we proposed a new interpretation of the human right to freedom of thought: N. Hertz, The Human Right to Freedom of Thought – Operationalising a Disputed Right in the Context of Neurotechnologies, Human Rights Law Review 25, 3, 2025.

Another central part of our work is analysing existing technology regulation of the European Union, in particular the new Artificial Intelligence Act. We published already two papers discussing different implications of the AI Act, one paper highlighting potential gaps and discussing possible solutions, such as permit procedures for high-risk AI: Voeneky, S., Key Elements of Responsible Artificial Intelligence: Human Rights, the EU AI Act, and the Need for AdaptiveInternational AI Regulation, in M. Löwisch, Th. Würtenberger, M.E. Geis, D. Heckmann (Hrsg.), Künstliche Intelligenz in Forschung, Lehre und Hochschule, Kap. 2, (March 2025); D. Feuerstack, D. Becker, N. Hertz, Die Entwürfe des EU-Parlaments und der EU-Kommission für eine KI-Verordnung im Vergleich. Eine Bewertung mit Fokus auf Regeln zu Transparenz, Forschungsfreiheit, Manipulation und Emotionserkennung, ZfDR 4/2023, 421

Continue to Part D