What would actually reduce AI risk
11 priorities for surviving the AI transition
Some say the only option is an indefinite international pause. Others say the US should accelerate to win the race against China.
Some say we need a single “Manhattan project” to prevent a corporate race. Others say we need to open source everything, or we’ll end up with the United States of Google.
These debates need to happen, but when it comes to taking action, ideally your plan doesn’t rest on such contested positions, or involve taking one side of a zero-sum competition.
Instead, given the stakes and uncertainty, I prefer to look for interventions that help (or are at least roughly neutral) according to many perspectives. Then I try to identify which are most neglected relative to their upside, especially in the most dangerous scenarios.
That’s what this article does. It’s a list of things we need to make happen to reduce the most pressing AI risks, especially loss of control, concentration of power and engineered pandemics, based on looking for points of agreement among the people I most respect in the space.
There isn’t a single solution that guarantees we’ll make it through, but there are many things we can do to make the world a bit safer. Collectively they could greatly increase the chance humanity navigates this wild transition.
Here’s a summary:
Accelerate alignment & control research before AI accelerates.
Make it clear what AI can already do with third-party testing and compute tracking, so no one can start an intelligence explosion in secret.
Draw red lines and plan emergency responses including an off-switch.
Prepare for a strategic slow down by hashing out how it would work now, so it’s ready when political will arrives.
Maintain a balance of power so that no single actor, company or country can dominate the future, while working towards a ‘grand bargain’.
Lock down the labs so frontier models can’t be easily stolen.
Eliminate the worst risks from engineered pandemics with wastewater monitoring, better PPE, air filtration and rapid vaccines.
Prepare for post-AGI challenges such as digital minds, space governance, and how to improve already good futures.
Prepare to use AI itself for all of the above since it may be the only way to respond fast enough during an intelligence explosion.
Wake up the world, then level it up by mobilising and training more people to work on these interventions.
Improve this list, since no-one can be confident what they’re doing is best.
1. Accelerate progress on AI alignment & control as much as possible, before AI itself accelerates
Loss of control of sufficiently powerful autonomous AI is a mistake that can only be made once. Before we reach an algorithmic feedback loop that starts an acceleration, we need (at a minimum) confidence the systems of that time are aligned, since those will be used to build and align the next generation. In case despite our best efforts the systems do end up misaligned, we want better ‘AI control’ methods to notice and prevent that behaviour. Ideally we’d have much better ideas for how to align superhuman systems. This all requires far more effort.
There are more and more concrete projects that can help, and the bottleneck is rapidly becoming engineering and operations. We can develop better tests for misaligned behaviour and get them implemented, study how AI models work so we can better predict when they’re honest, figure out how to use many weaker AIs to monitor a stronger one, and so on. It may also be possible to develop narrower tool AIs that can do medical research but don’t pose the loss of control risks. See list of projects in AI alignment, determining and testing AI character, mechanistic interpretability, and AI control.
2. Make it clear what AI can already do
The frontier companies have no obligation to tell the government about their capabilities, which means they could start an intelligence explosion in secret, enabling 1-2 companies to get far ahead, or create dangerous capabilities with no oversight.
We could create a third-party standards body that would test the latest models and publish the results. There could be regulations requiring reporting of capabilities relating to AI R&D or autonomy. Better whistleblower protections and legal support can help surface problems the other methods miss. Finally, we could have a lot more efforts like Epoch AI and METR to gather better data on what’s happening.
3. Draw red lines and create emergency response plans, including an off-switch
You may be able to help key actors define and agree on ‘red lines’: dangerous behaviour that must be avoided, and if observed, would trigger a preplanned emergency response, whether that’s reporting to government, turning off the model, stronger safety standards, or a broader slow down.
Today it’s easy to get frog boiled – capabilities that would have been worrying a couple of years ago are quickly normalised. So even if voluntary, red lines can help people notice key thresholds being crossed, but they could also form a basis for future regulation. This could look like working on responsible scaling policies, work within government, or international diplomacy.
Actors can also plan emergency responses in advance, making them quicker to execute. In particular, if an autonomous AI system escaped to the internet, there’s currently no way to track all the compute it might be using, or turn that compute off quickly, but it’s technically possible to design chips to have this feature, as well as building technical systems to turn off compute.
4. Prepare for an international strategic slow down
AI is improving extremely rapidly, and could accelerate further in an intelligence explosion. It’s implausible that the optimal speed for this transition is as fast as economically possible (the current default). More time could be used to figure out all the other interventions on this list.
It’s unclear that a complete and indefinite pause will be feasible or even desirable, but just an extra year, especially if it happens at the point when an algorithmic feedback loop becomes possible, has a totally different cost-benefit. It could significantly reduce the risks while only creating a small delay, and be far more politically feasible.
Even better would be to spread out an intelligence explosion over ten years rather than the 3-12 months it might take by default.
While getting a significant slow down right now seems unlikely, as AI gets more powerful, and likely more unpopular, political will will increase. A sudden event, like an escaped AI, could spur government into action.
Today it’s helpful to hash out how such a slow down might work and start to build a coalition around it, so that it’s ready to implement when political will increases. In particular, a pause won’t be practical without processes for verifying that it’s being observed. It would need to include China, so efforts to build dialogues between Chinese and Western researchers are also valuable. China is also concerned about the risks, and is behind the US in the race, so has a greater incentive to agree to one.
5. Maintain a balance of power, while working towards a grand bargain
Even if alignment is solved, an outcome where one company, one government, or one AI system ends up with unchecked control could be extremely bad. Positive feedback loops mean that the projects furthest ahead can increase their lead over time. Then AI means a single company could end up with the workforce of an entire country today, 24/7 surveillance of every single person is possible, and it would be possible to make a robotic military absolutely loyal to one person. All of these factors make it easier to lock-in power than in the past. We need to prevent this on multiple levels.
First, the most powerful AI models need to be designed so they don’t follow the command of a single person, like the CEO of a single company. Internal use of frontier AI needs to be tracked. We need to develop methods for testing AI systems to uncover secret loyalties and poisoned training data.
Simply open-sourcing everything isn’t a solution because it eventually means open-sourcing the ability to create things like advanced bioweapons, but it’s not a good idea to have a single leading AI project either, since it could gain undue power and there’s no back-up if it goes off the rails.
Internationally, it would be helpful if middle powers, such as the UK, EU, Japan, Australia, and South Korea, built up their AI expertise, so they’re able to vet American models, and put pressure on the US, such as demanding greater safety measures. This could look like efforts to wake up these governments, as well as build up the compute supply chain and datacentres, or even a frontier lab, within these countries.
If countries besides the US perceive they’re about to be permanently left behind by an AI boom, that would destabilise the international order, and perhaps trigger a pre-emptive war. This could be prevented by creating a treaty – a grand bargain – to share the benefits of AI in a way that each country will accept, in exchange for not trying to rush ahead of the others. Likewise, each country needs to develop an internal plan to share the benefits to avoid unrest.
6. Lock down the labs
Anthropic’s new Mythos model uncovered security vulnerabilities in computer systems that had been unnoticed for years, including one that had survived 27 years of human review. This is a small taste of what’s to come.
If frontier models can be quickly stolen by bad actors, it’ll be hard to have any breathing space to implement safety measures, and mean that dangerous capabilities get rapidly diffused before society can adapt. It also means it’ll be easy for AI agents themselves to escape or hack other AI companies.
Cybersecurity at the frontier AI companies, however, is still relatively weak and could easily be broken by determined actors. It’s probably not possible to make the labs secure to a determined national adversary at this point, but improving security can still help reduce the number of actors who can hack the models, and how long it would take them. Working on this could involve improving security at the frontier companies, introducing regulation around this security, or defensive efforts within the national security apparatus.
7. Eliminate the worst risks from engineered pandemics (+ the common cold as a bonus)
Pandemics are already a serious risk, but if AI accelerates the rate of technological progress, dual-use advanced bioengineering capabilities could arrive and be democratised much sooner, and it’s possible to design viruses that are more transmissible and deadly than naturally occurring ones. It also seems possible to design bioagents that persist in the environment, but would be deadly to humans, such as mirror life. Governments might develop these as a form of mutually assured destruction, or in the course of ordinary biology research, but there’s been a long history of lab leaks, so these could get released accidentally (if not by a group intent on destruction).
However, there’s a lot we can do to reduce the most serious risks (and regular respiratory viruses too). By regularly sequencing all the genetic material in wastewater, any material that’s growing exponentially could be flagged as a pandemic-in-waiting, giving us an early warning signal even for entirely novel viruses. Governments could build large stockpiles of (ideally much improved) personal protective equipment (PPE) so that, when a new pandemic is spotted, it can be quickly distributed to all essential workers, ensuring that society continues to function. Buildings could be retrofitted with UV lights that sterilise the air, and positive pressure air filtration. With enough measures to break transmission, the replication rate will drop below one and the pandemic will start to die out. State-of-the-art mRNA vaccines could then be rolled out quickly to prevent its return.
This requires engineers and entrepreneurs to build this hardware, as well as philanthropists and government to fund it. This work is unusually tractable and much of it is still neglected.
8. Prepare for post-AGI challenges and a better future
The first priority is to navigate the transition without loss of control of AI, or concentration of power into a narrow group, or other catastrophic mis-use like a pandemic. However, if we make it through, society still faces enormous questions: how do we treat digital minds, govern the expansion into space, or prevent ‘s-risks’ or gradual disempowerment? And even among good futures, there might be some that are far better than others.
If the rate of technological progress accelerates, we might be facing these questions much sooner than we expect. Work that’s done today might set precedent or otherwise influence the decision-makers who determine these issues, but today all of these questions are extremely neglected, with just tens of people working on them worldwide.
Many of the interventions above, such as preventing concentration of power and slowing an intelligence explosion, would also help. But I’d also like to see a lot more research to figure out what we might do today that could affect these issues, and developing ideas for regulation to make them more likely to be handled well, like an institute for space governance.
9. Prepare to use AI to help with all the above
Before we reach superintelligence, we’ll reach human-level systems, and before we reach those, we’ll have systems that are very useful. We should be planning to use these to navigate all the challenges above. Indeed, if the pace of change accelerates, it may be the only way to respond quickly enough. And if AI alignment turns out to be too difficult for humanity today, using cognitive labour from AIs might be the only way to solve it in time.
This could mean preparing to automate alignment research; creating AI tools to help key decision-makers; or preparing to use AI agents to figure out other items on this list, such as using AI for defensive cybersecurity and pandemic monitoring, or doing macrostrategy research. It also means securing more compute, model access and financial capital to be put towards these ends, and making sure it could be deployed at very short notice at the start of an acceleration.
10. Wake up the world, then level it up
Although people talk about AI all the time, the level of understanding of what’s happening remains low. Most implicitly treat it like a normal technology, rather than something that could jump start an intelligence explosion, industrial explosion, or even create a second species. And most of the people who are willing to actually act on that basis are working to accelerate AI’s already extremely fast rate of progress, which just gives us even less time to prepare. The number of people actually working full-time on the interventions is in the thousands (compared to 100,000+ working to accelerate AI’s arrival).
We need to wake up the world to these risks and make sure people understand what is actually happening. Then we need to train them in relevant skills, and help them take action. An especially pressing bottleneck is people in government who have a technical understanding of the technology.
This could look like mass communication, outreach to policymakers, community building among your own network, fundraising, running training programmes to help people enter the field and be more effective, like BlueDot, 80,000 Hours, MATS or the Horizon fellowship. These can be a multiplier on everything listed above. Read more.
11. Improve this list
What to do about AI is highly contested. Every intervention has drawbacks and could easily make things worse rather than better. Anyone who’s confident what they’re doing is helpful is wrong.
This means we could use far more research, debate and information to figure out what should be on this list, and make each item more concrete. This could look like big picture macrostrategy research, policy research, or research into any of the priorities listed.
How can you help with these priorities?
I hope to write more about this, but here’s the short answer: Try to work at an organisation in a role working at one of these priorities. This involves a wide range of roles, including management, comms, operations, and legal as well as technical or policy roles. (And here’s a longer list of career paths.)
If you don’t want to change job right now, anyone can also help by advocating for these priorities (even if just among your network), donating to these organisations, or helping others to pursue these jobs. You can also focus on investing in your skills and personal development so you’re in a better position to switch in the future.
If you just want to read one extra thing, here’s how to transition to work on AI risk in 3 months.