Deadline: 17-Aug-25
The Safeguarded AI programme is seeking a single founding team or entity to lead Technical Area 2 (TA2) Phase 2, focusing on the machine learning (ML) elements required to integrate frontier AI capabilities into a secure, general-purpose Safeguarded AI workflow.
Objectives
- TA2 will explore leveraging securely-boxed AI to train autonomous control systems that can be verified against mathematical models, improving performance and robustness. The workflow will involve forking and fine-tuning mainstream pre-trained frontier AI models to create verifiably safeguarded AI solutions. Key objectives of TA2 include:
- World-modelling ML (TA2(a)): Develop formal representations of human knowledge, enabling explicit reasoning and uncertainty accounting, to create auditable and predictive mathematical models.
- Coherent reasoning ML (TA2(b)): Implement efficient reasoning methods, such as amortised inference or neural network-guided algorithms, to derive reliable conclusions from world models. Safety verification ML (TA2(c)): Create mechanisms to verify the safety of actions and plans against safety specifications, using techniques like proof certificates or probabilistic bounds.
- Policy training (TA2(d)): Train agent policies that balance task performance with finite-horizon safety guarantees, including backup policies for safety failure scenarios.
Funding Information
- Total funding Phase 1 (total) £1m (to up to 5 team)
- Total funding Phase 2 (total) £18m (single award)
Eligibility Criteria
- They welcome applications from exceptional and ambitious researchers, organisational leaders, or experienced founders driven to create an alternative R&D pathway for safe and transformative AI.
- With respect to the entity which will ultimately deliver the TA2 research agenda, necessary requirements are:
- Based in the United Kingdom
- Credible ability to source world-class talent in machine learning research & engineering
- Robust governance mechanisms, including (among others) a diverse board with the sole mission of ensuring that decisions concerning the development, deployment and release of its AI technologies including algorithms, models, code, products or API access are made in service of humanity and society at large
- World-class cybersecurity
- Flexibility to pursue multilateral information-sharing and strategic partnerships with other private and/or government-sponsored entities if and only if determined to align with the mission
- In addition to ARIAs standard eligibility criteria here, the following types of entities are not eligible for funding to deliver TA2 2:
- For-profit companies
- Universities directly hosting TA2
- Based on these eligibility criteria, a non-exhaustive list types of applicants eligible for a TA2 award includes:
- New founding teams with a credible skillset and interested in quickly establishing a new UK-based non-profit institution from the ground up;
- Leading AI companies willing to create a UK-based affiliated 3 non-profit entities to host the TA2 R&D agenda, expanding the market for their AI capabilities into multiple critical infrastructure sectors;
- Established companies with existing critical-infrastructure businesses willing to create a UK-based affiliated 3 non-profit entities to become a pioneering supplier of guaranteed-safe AI capabilities;
- Established academic institutions willing to create, or partner in creating, a new UK-based affiliated 3 non-profit entities 4, where TA2 R&D can be pursued under conditions of first-of-class information- and cyber-security.
- The entity that will host the TA2 R&D agenda will be based in the UK. Non-UK citizens are welcome to apply to pursue this work, but need to be prepared to relocate to the UK.
For more information, visit ARIA.
