Deadline: 01-Mar-2024
The Future of Life Institute (FLI) is calling for research proposals with the aim of designing trustworthy global governance mechanisms or institutions that can help stabilise a future with 0, 1, or more AGI projects.
This current request for proposals is part of FLI’s Futures program, which aims to guide humanity towards the beneficial outcomes made possible by transformative technologies. The program seeks to engage a diverse group of stakeholders from different professions, communities, and regions to shape their shared future together.
These proposals should outline the specifications needed to reach or preserve a secure world, taking into account the myriad threats posed by advanced AI. Moreover, they expect proposals to specify and justify whether global stability is achieved by banning the creation of all AGIs, enabling just one AGI system and using it to prevent the creation of more, or creating several systems to improve global stability. There is the option for proposals to focus on a mechanism not dependent on a particular scenario, or one flexible enough to adapt to 0, 1 or more AGI projects. Nonetheless, it is recommended that applicants consider in which scenario their proposed mechanism would best perform, or be most valued. In that sense, as well as pitching a particular control mechanism, each proposal is also making a case for how humanity is kept safe in a specific future scenario.
FLI’s rationale for launching this request for proposal
- Reaching a future stable state may require restricting AI development such that the world has a.) no AGI projects; b.) a single, global AGI project, or c.) multiple AGI projects. By AGI, they refer to Shane Legg’s definition of ‘ASI’: a system which outperforms 100% of humans at a wide range of non-physical tasks, including metacognitive abilities like learning new skills.
- No AGI projects
- As recently argued by the Future of Life Institute’s Executive Director, Anthony Aguirre, if there is no way to make AGI – or superhuman general-purpose AI – safe, loyal, and beneficial, then they should not go ahead with it. In practice, that means that until they have a way of proving that an AGI project will not, upon completion or release, take control away from humanity or cause a catastrophe, AGI models should be prevented from being run.
- Proposals focusing on such initiatives might include mechanisms for:
- Risk analysis;
- Predicting, estimating, or evaluating the additive benefit of continuing AI capability development;
- Ensuring no parties are cheating on AGI capability development;
- Building trust in a global AI governance entity; or,
- Leveraging the power of systems to solve bespoke problems that can yield societal and economic benefits.
- One AGI project
- As Aguirre points out, even if an AGI is ‘somehow, made both perfectly loyal/subservient to, and a perfect delegate for, some operator’, that operator will rapidly acquire far too much power – certainly too much for the comfort of existing powers. He notes that other power structures ‘will correctly think… that this is an existential threat to their existence as a power structure – and perhaps even their existence period (given that they may not be assured that the system is in fact under control.)’ They may try to destroy such a capability. To avoid this imbalance or its disastrous results, the single AGI will either need to be created by a pre-existing cooperation of the great powers, or brought under the control of a new global institution right after it is produced.
- Multiple AGI projects
- Similar concerns arise if multiple AGI systems emerge. A delicate balance of power must be strenuously maintained. Furthermore, with all of these scenarios there will ensue the significant dual problem of on the one hand limiting the associated risks of such powerful AI systems, and, on the other, distributing the associated benefits – of which they can expect many.
- Flexible to different numbers of AGI projects
- Dividing scenarios into the three above groups will hopefully yield a balanced sample of each outcome. Equally, urging primary investigators to select just one of these categories may help to encourage concrete scenarios for what a well-managed future with advanced AI could look like – in short, push them to pick only their preferred future. However, some applicants may wish to submit proposals for mechanisms flexible enough to adjust to varying numbers of AGI systems. These applicants will need to address how their mechanism can ensure stability in a world where the number of AGI systems can keep changing. Alternatively, there may be mechanisms or institutions proposed whose function is not dependent on a particular number of AGI projects. In such cases, it is still recommended that applicants consider in which scenario their proposed mechanism would best perform, or be most valued; nonetheless, they leave this option available for those cases where such a consideration proves to be unenlightening.
- Existing projects
- FLI is by no means creating a new field here. Promising initiatives already in the works, which may inspire researchers applying to this program, include the following:
- FLI’s international AI governance organization proposal
- Davidad’s Open Agency Architecture
- Robert Trager’s International AI Organization
- Conjecture’s Multinational AGI Consortium (MAGIC) – a ‘CERN for AI’
- FLI is by no means creating a new field here. Promising initiatives already in the works, which may inspire researchers applying to this program, include the following:
- No AGI projects
Levels
- Level 0: No AI
- Level 1: Emerging
- equal to or somewhat better than an unskilled human
- Level 2: Competent
- at least 50th percentile of skilled adults
- Level 3: Expert
- at least 90th percentile of skilled adults
- Level 4: Virtuoso
- at least 99th percentile of skilled adults
- Level 5: Superhuman
- outperforms 100% of humans
Evaluation Criteria & Project Eligibility
- Proposals will be evaluated according to the track record of the researcher, the proposal’s originality or potential to be transformative, the potential for the proposed activity to advance knowledge of the coordination problems for mitigating AGI risk, and how convincingly the proposal accounts for the range of AGI risks.
- Grants applications will be subject to a competitive process of external and confidential peer review. They intend to support several proposals. Accepted proposals will receive a one-time grant of $15,000, to be used at the researcher’s discretion. Grants will be made to nonprofit organizations, with institutional overhead or indirect costs not exceeding 15%.
For more information, visit FLI.