In case you’re new(!), the idea is that your career is your most important decision, especially for your impact on the world. But most career advice sucks, so millions waste it. They drift into the standard defaults.

I spent the last 15 years of my career researching how to find the best career, and the book sums up everything I’ve learned.

I argue the route to a fulfilling career isn’t to follow your passion, it’s to build valuable skills and apply them to the most pressing problems of our time. The book unpacks exactly how.

We hit number 1 in our category in both Amazon US and UK. (There’s still two more days for sales for to count for NYT 😬)

Rutger Bregman, author of Utopia for Realists, said it was one of the only self-help books he’d recommend(!):

“For young graduates who’d like to avoid a soul-crushing career in the Bermuda Triangle of Talent (consulting, finance and corporate law), this book is a lifesaver…research-backed, free of jargon, and pretty funny as well!”

Andy Masley, who has single-handedly brought sanity to the discourse on datacenter resource consumption in the last year, said:

“The original book is still one of the great holy texts for me, partly because the basic message is so simple but so not acted on almost anywhere: you should actually try to think a lot about planning your career to do good in a really quantitative way. Once you start to pay attention to how many people’s careers basically involve coasting along on the first thing that gave them status, the basic ritual of actually writing down and thinking through what you’re doing looks a lot more important.”

My favourite review of all was by Bentham’s Bulldog.

I was especially touched to hear stories of people who the guide has helped in the past. Sneha Revanur, founder of Encode AI, the US’s leading AI youth movement, said:

“My career, and the rebirth of Encode, are entirely downstream of Ben and his team. Couldn’t recommend this book more.”

The CEO of the UK’s leading catastrophic risk think tank, Angus Mercer, said it had helped him find a more impactful career. I even learned that the general manager of the Dwarkesh Podcast, Max Farrens, wouldn’t be in the job without the guide.

Cate Hall, author of You Can Just Do Things, and person with the most insane CV I know, said she wished the book had been around when she graduated to stop her from going into law. Hannah Ritchie from Our World in Data said it’s where she sends people for career advice.

One of the funnest parts of writing the book was working with a real new yorker cartoonist, Natalya Lobanova, to illustrate it. She posted her favourite cartoon on instagram, getting 10x more likes than anything we’ve posted there.

Some of the ideas people have been most interested in so far:

• Still the world’s most underrated chart: the best charities save lives 10,000% more cheaply than typical.

• The best and worst predictors of job performance.

• With tens of billions of dollars of extra philanthropy on the way, now more than ever we need people working on these problems.

• The classic 5 best predictors of job satisfaction.

• And also my meditation advice!

The last few weeks have been intense, and I was also interviewed on the 80,000 Hours podcast about what short AI timelines mean for your career (YouTube).

I was also on The Cognitive Revolution getting into more practical detail about what people can do about AI risk. And released a video essay about the most valuable skills of the future.

There will be an upcoming interview on EconTalk, one of the original great podcasts I’ve listened to for 10+ years, talking about the classic ideas of the book. (Plus a popular interview on Vox.)

Tomorrow, I’ll be doing an AMA on r/IAmA at 8am pacific, 11am eastern and 4pm UK. Check my socials for the link.

If you’d like to help us get even more people tackling the most pressing problems:

• There’s still two days left to help us hit the NYT bestseller list – buy a physical copy before midnight 30th May!

• We lost hundreds of old Amazon reviews, so our page doesn’t look great. If you’ve read the guide in the past, consider leaving a review.

Thank you again so much!

Ben

This was true ten years ago, but as we stand on the brink of human-level AI, I’ll argue it’s even more true today.

You’ll spend about 80,000 hours at work over your life. Unless you happen to be the heir to a large estate, it’s the biggest resource you have to make a difference.

That means even very small improvements to your career matter a great deal. If you can increase the overall impact of your career by just 1%, it would be worth spending up to 800 hours working out how to do that.

And it’s possible to increase the impact of your career by much more than 1%. And by far more than people realise.

When you poll people about how much more effective they think the best development charities are at saving lives compared to the average, they guess they’re about 50% better. A noticeable difference, but not huge.

Poll experts in global health, however, and they’ll say the best are around 100 times more cost effective, a difference of 10,000%.

In short, there are huge differences in the impact of different ways of helping people, but no one knows about them.

And that’s within a single area looking backwards.

We may now be on the brink of human-level AI. In the next three to ten years, we face risks from AI that could kill a large fraction of people, such as AI-enabled bioweapons, or others that could permanently disempower humanity, such as concentration of power into a small minority of companies, or loss of control of autonomous AI agents.

These risks only have a few thousand people working on them at best, making them over 100 times more neglected than global health.

People joke we may now only have 8,000 hours left in our careers. But if true, these are the years in which your choices matter most. They’re your chance to help humanity navigate its most dangerous moment.

The personal stakes have increased too. Gen Z are more pessimistic than ever, and over half of people are worried their job will be automated. They’re right to be concerned. The traditional answers — law, medicine, consulting — no longer obviously make sense like they once did. But AI tools have also made it possible for teams of three to accomplish what would have taken thirty before, and the value of the skills needed to do that are increasing dramatically. As the pace of change accelerates, small differences in your decisions get magnified.

In general, careers differ in impact for three main reasons:

1. Some problems are far bigger and more neglected. Most people who want to do good still focus on social issues in their home country, but today the world faces truly existential risks with hardly anyone working on them. (More.)

2. Some paths let you contribute far more. Most altruistic people focus on traditional helping careers, but unknown government bureaucrats, executive assistants, and even podcasters can often affect far more people. (More.)

3. Some careers give you higher chances of outsized success. A landmark study of leaders across a range of fields found that about half of the contributions were made by the top 10% of the performers. (More.)

These three factors multiply together rather than merely add. If you can eventually find a problem that’s twice as big or neglected, make twice as large a contribution to it, and find a path where your chances of success are twice as high, then (holding all else equal) you can expect to have eight times as much impact.

In practice, I think it’s often achievable to have 10 or 100 times as much impact as what you would have done otherwise, and sometimes 1000 times more.

It’s easy to gloss over these differences. Intuitively, people often group careers into those that “help” (e.g. doctor, charity worker, teacher), those that are neutral (e.g. accountant), and those that are unethical (e.g. oil baron). But that’s a huge mistake.

If among careers that are “impactful” you can find an option that’s 100 times more impactful again, then 10 years doing that could achieve what would have taken 1,000 years otherwise. You could spend the remaining 30 years of your career meditating on the beach and still have done far more good for the world.

If you can’t change job right now, finding a way to contribute indirectly through your current role, such as by donating, spreading ideas or community building, can also often achieve than standard helping careers, provided you target that effort at the right issues. (And if you need to take care of yourself and immediate family right now, do that first.)

What might this impact look like more concretely? I’ve argued that anyone capable of earning a graduate salary in a high-income country can save 80 lives over their career – and that’s a lower bound, and without changing job. If you have the privilege be able to switch paths and work on something unconventional, your impact should be a lot higher.

I’d also argue it’s often better for you personally. A sense of meaning is a big part of life satisfaction, and pursuing it makes you more resilient, more likely to get help, and more likely to build skills that the world actually values. So much career advice is self-focused, telling you to follow your passions and make money. But people do both and still feel empty at the end. This isn’t to say it’s easy to work on the world’s problems, but it’s worth trying.

When you look back on this time in history, how will you wish you’d responded?

All this is why I wrote 80,000 Hours. I believe many people can find work that’s both more rewarding and more impactful than the standard defaults.

And as the world faces one of its most dangerous moments, now more than ever, we need people trying to tackle the world’s biggest problems, rather than making PowerPoints at PwC.

The book aims to be a complete guide to how. I’ve tried to make it the most research-backed and forward-looking career guide ever written. It’s the culmination of 15 years of thinking about these questions.

It’s out now, and is available anywhere you can buy books (inc. an audiobook, narrated by yours truly).

I’m deeply grateful for any shares on social, Amazon reviews or recommendations.

To help us hit the bestseller lists, buy before the 30th from an independent store.

Or here’s a link to Amazon.

Get the book

Thank you!

The answer involves looking for problems that are not only big, but also solvable and neglected, because that’s where an extra person can have the greatest impact.

Although this framework sounds simple, it has led me and 80,000 Hours to some radical places. This article sums up the entire journey, explaining:

• Why we initially focused on global health, and how preventing diarrhoea has saved as many lives as world peace.

• Why we recommended trying to prevent the next pandemic long before COVID-19, and tackling AI risk long before ChatGPT.

• Which problems we think are most pressing today, and where we might focus in the future.

It’s adapted from Chapter 5 of my book. It was originally published on 80,000 Hours. It’s a long read, but it compresses all the most important things we’ve learned about this question over the last 15 years.

Why charity doesn’t begin at home

Most people who want to do good focus on the problems they see all around them. In rich countries, this often means issues like homelessness, inner-city education, and unemployment. But, while a natural starting point, are these really the most urgent issues?

In the US, only 5% of charitable donations are spent on international issues,¹ while the large majority is spent explicitly on domestic ones. Around 45% of US college graduates enter careers in education, health, and public administration — which mainly involve helping people at home in the US.² (Most of the remainder take corporate jobs, and it’s similar in other countries.)

There are some good reasons to focus on helping your own country — you know more about the issues, and you might feel you have special obligations to it. However, back in 2009, we encountered the following series of facts which changed our minds.

First, consider the distribution of world income:

Even someone living on the US poverty line ($15,650 per year, as of 2025) is richer than about 85% of the world’s population and about 20 times wealthier than the world’s poorest 800 million, most of whom live in Central America, Africa, and South Asia and earn about $1,000 per year.³

Not only are the poorest people in the world much poorer than those in the US, there’s also a lot more of them. There are about 40 million people living in relative poverty in the US, just 5% of the 800 million living in extreme poverty globally.⁴

Crucially, there are far more resources dedicated to helping this much smaller number of people. Overseas development aid from all countries is under $200 billion per year, compared to $1.7 trillion spent on welfare in the US alone.⁵ This is what we’d expect given the biases we covered earlier.

We also learned that a significant fraction of US social interventions probably don’t work at all. This is exactly what we’d expect based on the difference in wealth. If a problem in the US persists, it’s probably because it’s complex and can’t be easily solved with more resources. By contrast, the world’s poorest regularly die from things like contaminated water, which almost never happens in the US.

This isn’t to deny that the poor people in rich countries have tough lives, perhaps even worse in some ways than those elsewhere. Rather, the issue is that there are far fewer of them, and they’re much harder to help. And this argument can be extended to other rich countries, including the UK, Australia, Canada, and most of the EU. This raises the question, what are the most pressing problems facing the world’s poorest people?

Earlier, we told the story of Dr Nalin, the pioneering physiologist who helped to develop oral rehydration therapy as a treatment for diarrhoea.

As a result of this kind of work, the number of deaths each year due to diarrhoea has fallen by 3 million over the last five decades. And there have been similar victories over other infectious diseases. Meanwhile, wars and political famines killed an average of two million people per year over the 20th century.⁶ So we could say that efforts by Nalin and others did more to save lives than achieving world peace would have done.

The global fight against disease has been one of humanity’s greatest achievements, but it’s an ongoing battle, and one that you can contribute to with your career. Many of these victories — such as the vaccination drive that eradicated smallpox — were driven in part by international humanitarian aid, and there is more to be done.⁷

Now consider the following data. It’s the data Toby Ord introduced me to, and which eventually led me to found 80,000 Hours.

The graph shows health treatments, such as tuberculosis medicine or cataracts surgery, in order of how much ill health they reduce per dollar, as measured in randomised controlled trials. Ill health is measured using a standard unit used by health economists called the ‘quality-adjusted life year‘ (QALY) that takes into account both the severity of the disease being treated, and how many years of benefit are provided.

All the treatments studied are effective, and all of them would be funded in North America or Europe.

For instance, drugs to treat the AIDS-related cancer Kaposi’s sarcoma are expensive and only provide a small benefit. This puts them around the threshold of what would get funded in a rich country (such as via health insurance), but are still seen as clearly worth funding. However, the data found that antiviral therapy would, for the same cost, prevent AIDS in a much larger number of people, preventing over 10 times as much ill health. Preventing the transmission of HIV during pregnancy in the first place, meanwhile, is another four times cheaper again.

The very most effective interventions in the entire sample, like childhood vaccinations, were about 10 times more cost effective than the mean, 60 times the median, and 15,000 times more than the worst.⁸

This is an astonishing result — perhaps suspiciously so. It turned out there were mistakes in the original analysis that meant the effectiveness of the top result was overstated.⁹ But even after these mistakes were corrected, the top interventions remained at the top, and were still much more effective than average.

This means if you were to work at a health charity focused on one of the most effective interventions, you could expect to have at least several times more impact compared to a randomly selected one, and probably 100 times more impact than those in the bottom half.

How much more impact might you be able to make in your career by switching your focus to global health? As we’ve seen, because the world’s poorest people are over 20 times poorer than the poorest in rich countries, resources should go about 20 times as far in helping them.¹⁰ We can then also use the data above to pick the very best interventions within global health, allowing us to have perhaps five times as much impact again.¹¹ Those combine to make a 100-fold increase in expected impact.¹²

Does this check out? The UK’s National Health Service (NHS) and many US government agencies are willing to spend over $30,000 to give someone an extra year of healthy life, or over $1 million to save a life.¹³ This is a fantastic use of resources by ordinary standards. However, as we saw earlier, the nonprofit GiveWell has identified real charities that can use a $3,000 donation to save a child’s life. This is about 0.3% of what it typically takes to save a life in a rich country.

So a year spent working somewhere like the Malaria Consortium might improve health as much as working in a typical healthcare job in a rich country for 300 years.¹⁴

These discoveries caused many of us at 80,000 Hours to start giving at least 10% of our incomes to effective global health charities. No matter which job we ended up in, these donations would enable us to make a significant difference. In fact, if the 100-fold figure is correct, a 10% donation would be equivalent to donating 1,000% of our income to charities focused on poverty in rich countries. See more detail on how to contribute to global health in our full profile.

However, everything we learned about global health raised many more questions. If it was possible to have 10 or 100 times more impact than the most common ways of helping others, perhaps with a bit more effort, we could find something even better?

Where might an even greater scale of suffering be found?

Are there any global issues that cause even more suffering than global poverty? One answer to this question was put forward by Australian moral philosopher Peter Singer.¹⁵

Around a trillion animals die every year in factory farms in conditions that would be considered torture if inflicted on your pet.¹⁶ Chickens are bred to grow so fast they can barely walk and are kept in tiny cages their entire lives. Female pigs are forced to lie in crates so narrow they can’t even roll over, before dying painfully in gas chambers.¹⁷ Fish are killed by leaving them to suffocate for hours in the open air.

Over 97% of the meat eaten by humans is produced in factory farms, with genuinely ‘high welfare’ meat forming a tiny, tiny minority.¹⁸ Animals in factory farms have no economic or political power. They depend entirely on our compassion — and because they are out of sight, they get almost none.

Most philanthropic efforts dedicated to helping animals are directed towards things like The Donkey Sanctuary, which is one of the UK’s best-funded animal charities.¹⁹ It aims to give working donkeys a comfortable retirement, attracting huge bequests from pensioners who like cute animals.

The entire philanthropic field dedicated to ending factory farming, by contrast, receives about $400 million per year, only 0.03% of total philanthropic funding in the US.²⁰ That’s under 1% of donations to international development (which in turn receives only 5% of the total).²¹

Making the comparison between helping humans and preventing animal suffering is philosophically controversial, but almost everyone agrees that it’s bad to torture animals. And it also turns out this suffering can be reduced extremely cheaply.

There’s a long-standing belief that the best way to stop factory farming is to convince people to stop eating meat. But that doesn’t appear to work — at least not anymore. Over the past 20 years of animal advocacy, the number of vegans and vegetarians has basically stayed flat.²² This makes sense: meat is delicious, widely available, and plays a central role in many cherished cultural traditions. Moreover, most people are already aware of the arguments for vegetarianism, and yet they haven’t changed.

In contrast, recent efforts to convince companies to switch from caged to cage-free eggs have been enormously successful. One philanthropically funded campaign cost around $85 million over 10 years, but persuaded hundreds of companies in the US and EU to make the switch. This increased the cost of an egg by only $0.01–0.03, but has already saved around 1 billion chickens from living agonising lives in cramped cages.²³

There are many more welfare reform campaigns that could be run. It’s too simplistic to say that aiming to bring about ‘institutional change’ is always better than trying to change individual behaviour, but this is a case where it is.

Another approach is the development of cheap, tasty substitutes, whether they be plant-based meat (like Beyond Burgers) or cultivated meat grown from a sample cell that is physically identical to animal meat. These strategies aren’t always profitable, but with subsidies it could be possible to drive down costs and develop a self-sustaining industry — in the same way subsidies were needed to develop solar panels, which are now cheaper in many places than fossil fuels (which in turn benefit from huge subsidies). Reducing meat consumption would also be great for the planet, as agriculture is one of the biggest sources of greenhouse gas emissions.²⁴

One individual we worked with, Richard, was working in international development policy, helping to ensure that aid spending was focused on the most cost-effective programmes. He wasn’t an ‘animal person’: he didn’t have a pet, and didn’t find farmed animals especially cute or appealing. But after learning about the vast number of animals in factory farms, the cruelty of their treatment, and the tiny amount of resources dedicated to helping them, he became convinced that he should shift his focus.

After visiting a factory farm and slaughterhouse, and being shocked at the violence and suffering he saw, Richard joined the Good Food Institute (GFI), an NGO that provides research and policy advice aimed at kick-starting the alternative proteins industry.

GFI has supported over 100 new projects and helped secure £27 million of funding for alternative proteins from the UK government, as well as €38 million from the German government. Richard and his team also helped to defeat attempts to ban plant-based products from using ‘meaty’ names, such as ‘burger’ or ‘sausage,’ in the EU.

We still think factory farming is an urgent problem, as we explain in our full problem profile. We helped to found Animal Charity Evaluators, which does research into how to most effectively improve animal welfare. But while focusing on animals looks like one way to find problems that are even bigger and more neglected than global health, we wondered if we could find something even bigger again.

The importance of future generations

Imagine the following: you throw away some broken glass in the forest. Later, a child walks by and cuts their foot. Now, suppose the child only cuts their foot 100 years in the future. Does that mean it wasn’t bad to throw away the glass after all?

Most people would say no, which tells us that they care about future generations. In a similar way, it’s hard to understand why people would care about their grandchildren’s children, or their legacy in business, art, or science, or about preserving the natural world, if they didn’t care about what will happen after they’re gone.

This simple idea — that future generations matter — has some radical implications about where to focus when applied to the world today. We were first exposed to these implications by researchers at the University of Oxford’s (modestly named) Future of Humanity Institute. William MacAskill, my cofounder at 80,000 Hours, later helped to coin the term ‘longtermism,’ the view that helping future generations should be a key moral priority of our time.

Here’s the argument:

First, future generations matter, but they can’t vote, buy things, or stand up for their interests — much like animals in factory farms. This means our system neglects them. In addition, their plight is abstract. The suffering on factory farms is just a few clicks away on YouTube, whereas we can’t so easily visualise lost future potential. Future generations rely more than any other group on our goodwill, and yet even that is hard to muster.

What’s more, Earth will remain habitable for hundreds of millions of years at least, and while it’s true we may die out before that point, if there’s a chance we’ll make it, then we’d expect many more people will live in the future than are alive today.²⁵

To use some oversimplified figures: if there’s one generation per century, then over 100 million years there would be 1 million future generations.²⁶ This is such a big number that any problem which affects future generations is potentially of a far greater scale than an issue which only affects the present. It could end up affecting the lives of millions of times more people, with all the art, science, culture, joy, and suffering those lives will entail. So the problems that affect the future are not only likely to be neglected, they’re also potentially the largest in scale, nearly no matter what you value. (We cover these ideas in more depth in a separate article.)

The last, crucial point is that there are things we can do today that will help both people living now and future generations. What might those be?

The case for focusing on neglected existential risks

In the summer of 2013, Barack Obama referred to climate change as “the global threat of our time.” He’s not alone in this opinion. When people think of problems facing future generations, climate change is usually the first thing that comes to mind. Polls find that young people routinely rate it as the world’s most pressing issue.²⁷

One reason for that is many fear that climate change could lead to a catastrophic collapse of civilisation, and even the end of the human species.²⁸ One of the most famous climate advocacy groups, Extinction Rebellion, named itself in opposition to this possibility.

We think this is on the right track — but that the rebellion could be broadened. A strong candidate for the most effective way to help future generations is to prevent a catastrophe that ends civilisation, since such a catastrophe would prevent future generations from even existing.

So long as civilisation continues, however, there’s a good chance that problems like poverty and disease will eventually be solved. Anything that poses a truly existential threat, however, would prevent any such progress forever.²⁹

To illustrate the difference, consider two scenarios:

1. A nuclear war kills 99% of people, but civilisation recovers.

2. A nuclear war kills 100% of people.

If you only focus on the present, the second scenario is only about 1% worse than the first. However, if you factor future generations into the equation, then the second is much, much worse, since it eliminates all future potential as well.

From this perspective, we should pay a lot more attention to risks that are most likely to be existential (defined as those that risk permanent loss of civilisation’s future potential, whether via extinction or lock-in of a worse future).³⁰ Instead, people often mislabel risks as existential when they aren’t, while those that actually are get less attention. (See more on the argument for focusing on existential risks.)

This is where we disagree with President Obama, or more precisely with the widely held view that climate change is the world’s most pressing existential risk.

The Intergovernmental Panel on Climate Change’s (IPCC) “Sixth Assessment Report” is clear: climate change will be hugely destructive. Most likely we’ll see 2–3ºC of warming, and this will cause floods, famines, fires, and droughts. The world’s poorest people will be affected most.

But, even when we try to account for tail risks and other uncertainties, nothing in the IPCC’s report suggests that civilisation itself will be destroyed. The worst-case scenarios outlined in the report involve 6ºC of warming, and in those most of the Earth would still remain habitable.

Here’s one illustration:³¹ even with 5ºC of warming, wheat yields in temperate regions would most likely increase around 18%, due to the longer growing season. It’s true that yields of crops like maize in regions near the equator would fall about 24%, which could cause famine in those regions. But this comparison is between a worst-case future and what would have happened without climate change.

Since 1961, crop yields have steadily increased around 200% due to research and innovation, despite the 1–1.5ºC of warming we’ve already experienced. So, even if climate change causes yields to decline 30% by the end of the century, they’ll most likely end up far higher overall.³² Pessimistic analyses usually completely ignore the forces of innovation working in the opposite direction from climate change, as well as the many ways we could adapt.

Similarly, the most negative analyses of the economic cost of climate change argue it could reduce global GDP by 30%.³³ That’s a huge decline. But it fails to take into account ongoing economic growth of about 2.5% per year. If that continues, then in 75 years our descendants will be six times richer than we are. A 30% cut in GDP caused by climate change would mean they’ll only be 4.2 times richer. In fact, in all of the IPCC’s scenarios, it’s projected that average per capita income will be higher in 2100 than it is today.

Climate change is also already widely acknowledged as a major problem — conspiracy theories aside. Most governments have signed binding treaties, and as of 2024, philanthropic spending on climate change is around $6–10 billion per year, government grants run into tens of billions in the US alone,³⁴ and all financing for climate initiatives internationally runs to about $1.6 trillion.

In most rich countries, CO2 emissions have already fallen significantly, and the continued decline in the cost of solar panels, batteries, and electric vehicles means it will be possible to continue these trends.³⁵

So, while we think tackling climate change is an important way to help future generations — and you can read more about the risk from climate change in our full profile — we think there are much more neglected, and more existentially dangerous issues for you to consider working on.

Biorisk: the threat from new pandemics

In 2006, The Guardian newspaper ordered segments of smallpox DNA in the mail. If assembled into a complete strand and transmitted to 10 people, a study estimated the virus could infect up to 2.2 million people in 180 days — potentially killing 660,000 of them — if authorities did not respond quickly with vaccinations and quarantines.³⁶

We first wrote about the risks posed by catastrophic pandemics back in 2014.³⁷ Our reasoning was that major pandemics arise every few decades, but not much was being done to prepare for the next one. Six years after that, COVID-19 caused over $10 trillion of economic damage,³⁸ and most likely killed over 20 million people.

Despite all this damage, it’s unclear whether we’re any better prepared for the next one. Moreover, a future pandemic could easily be both more infectious and more lethal than COVID-19, for instance with a 10% fatality rather than 1%. A disease this dangerous could shut down global supply chains, resulting in food shortages and even a breakdown of law and order.

On top of this, each year bioengineering technology becomes cheaper, making it accessible to more and more people. What once required a state-of-the-art laboratory with 50 scientists working around the clock can be done by a lone hobbyist today.³⁹

Eventually, it will be possible to engineer pandemics that are much worse than those that have arisen naturally. Imagine someone released 10 different diseases with a three-month incubation period, the infectiousness of measles, and the fatality rate of Ebola. Almost everyone in the world would be infected before anyone had symptoms.

The world has plenty of religious cults, despots, and would-be school shooters who might decide they want to take everyone else down with them. A state like North Korea could decide to develop these kinds of bioweapons as deterrence to invasion. Mutually assured destruction became our policy with nuclear weapons, and so it could again with biological ones. The world would be one lab leak away from catastrophe.

Given what we know about the pace and accessibility of bioengineering tools, the chance that there will be a pandemic that kills over 100 million people during the next century seems similar to the risk of large-scale nuclear war or climate change above 6ºC.⁴⁰ An engineered pandemic could also kill over 90% of the population, suggesting its overall scale is significantly larger.⁴¹

But risks from pandemics are, even now, far more neglected than either of these. In comparison to $6–10 billion of philanthropic funding for climate change, and $1.6 trillion of total climate finance, pandemic prevention only receives $1 billion of philanthropic funding, and total spending aimed at reducing the chance of worst-case pandemics is probably under $10 billion.⁴²

At the same time, it’s an unusually solvable problem. By regularly sequencing all the genetic material in waste water, any material that’s growing exponentially could be flagged as a pandemic-in-waiting, giving us an early warning signal even for entirely novel viruses.

Governments could build large stockpiles of (ideally much improved) personal protective equipment (PPE) so that, when a new pandemic is spotted, it can be quickly distributed to all essential workers, ensuring that society continues to function. Buildings could be retrofitted with UV lights that sterilise the air. With enough measures to break transmission, the replication rate will drop below one and the pandemic will start to die out. State-of-the-art mRNA vaccines could then be rolled out quickly to prevent its return.

Overall, we think biosecurity is one of the world’s most pressing problems today (you can read more about how to contribute to biosecurity in our full profile). What’s more, you don’t need to be a biologist to make a difference. What’s most needed is people with skills in organisation-building and engineering, who can do things like develop and distribute cheaper PPE, and run waste-monitoring systems. There’s also a need for people working in government and policy to make sure biosecurity is properly funded and prioritised.⁴³ Read more about preventing catastrophic pandemics.

But could there be even bigger existential risks facing humanity?

Why AI could change everything (& even more than people think)

Around 1800, civilisation underwent one of the most profound shifts in human history: the Industrial Revolution.⁴⁴

When you look at wealth, population, the rate of technological progress, or even the rate of social change over time, basically nothing happened for many thousands of years, until suddenly everything began to accelerate.

Looking forward, what might be the next transition of this scale? Something that causes an even greater acceleration and shapes the lives of all future generations? If we could identify such a transition, it could well be the most important area to work on.

Back in 2016, we decided the most likely candidate was AI.⁴⁵ We’d been tracking the issue much earlier, but that was the year an artificial neural network mastered the game of Go — a Chinese board game requiring strategic intuition — much faster than expected.⁴⁶ Many AI researchers began predicting that human-level AI was likely to be developed in our lifetimes.

We reasoned that the arrival of computational systems smarter than humans would be one of the biggest events in history, akin to the arrival of a new, intelligent species.

Consider that chimpanzees are faster than us, and much stronger. But there are under 300,000 of them in the wild, compared to 8 billion humans, and they depend on us for their fate.⁴⁷ This is due to our tools, culture, and ability to cooperate and solve novel problems, which rest to a large extent on our greater intelligence.

AI could be unlike any previous technology due to its generality. Invent an axe, and you can cut things better. Invent an intelligent machine, and it can learn to do any task.

When people talk about ‘artificial general intelligence‘ (AGI), that’s what the ‘general’ part means. A generally intelligent AI can learn to do a wide range of tasks in the same way that humans can. In contrast, a ‘narrow’ AI can only do a small range of tasks, like play chess or calculate numbers. All past technologies have been narrow compared to human abilities.

Since 2016, many experts have been further surprised at how quickly AI has advanced, making the issue a lot more urgent. On the forecasting platform Metaculus, hundreds of forecasters predict how many years it’ll be until AGI is created. Since 2020, the median estimate has fallen from 50 years to five.

The definition of ‘general intelligence’ is hotly debated, but no matter the definition, all major forecasts have shown large declines . For instance, in a 2023 survey of thousands of AI researchers published in top-tier journals, the median estimate for when “high-level machine intelligence” would be created declined by 13 years compared to the same survey just one year earlier.⁴⁸

In less than five years, the large language models (LLMs) used to power products like ChatGPT, have gone from barely being able to string a few sentences together to conversing in natural language in a way that’s basically indistinguishable from humans. More recently, they’ve become able to answer known scientific questions better than PhD holders in the relevant subject,⁴⁹ to beat almost all human experts in coding competitions,⁵⁰ and to win gold at the International Mathematical Olympiad.⁵¹

This progress was driven by massive increases in the amount of computing power used to train AI models, which has grown by over four times per year since 2010. It’s also been driven by rapid algorithmic progress, which in turn has been driven by a rapidly growing AI research workforce. These trends seem likely to continue for at least the next few years, meaning we should expect further rapid AI progress during that time.

When people picture much more capable AI, they sometimes imagine speaking to an even smarter chatbot. But that’s not where we’re headed. AI companies are trying to create a ‘digital worker’ that you can ask to do open-ended projects, like build an app, run a sales campaign, or design a scientific experiment. We believe there’s a fair chance systems like these will exist by 2035.

Suppose systems like this are reached — what happens then? To many people, what first comes to mind is job loss, and we’ll discuss how not to lose your job to AI in part eight on the skills we believe will be most valuable in the future. But I think the consequences could be much wilder, and could arrive well before mass unemployment becomes a serious risk.

The theoretical possibility of feedback loops in AI development was identified at the dawn of the field by its founders, including Alan Turing and I. J. Good.⁵² The idea is that if AI itself can start to help with AI research, then progress will speed up, which would mean AI becomes even more advanced, which could speed up progress even more, and so on. But in the last five years, it’s become much clearer how such a feedback loop could work in practice.

The leading AI companies today are already using AI extensively to aid their own research. In particular, they use AI to help with coding, both because coding is what AI is best at today and because it’s a crucial part of doing AI research. And yet the overall boost to productivity remains relatively small.

Imagine, though, what would happen if AI was able to do the job of a junior engineer, and then a mid-level engineer, and continued to improve.⁵³ If current models were able to produce work comparable to that of a mid-level engineer, then given the amount of computing power already available in data centres, it would become possible to have the equivalent of millions of competent engineers working on AI research.⁵⁴ As AI continues to improve, eventually these models could start to do the work of even top researchers.

In comparison, there’s probably under 10,000 human researchers working on frontier AI today, so the workforce would, in effect, expand in size more than 100 times. No-one knows exactly how much that would speed up progress. The most careful estimate I’ve seen is by Tom Davidson, who currently works at Forethought — a research group Will MacAskill established in Oxford to explore the impact of AI on society. Tom estimates we’d most likely get three years of AI progress in one year, and it’s possible we’d see 10.

Over the last five years, improving algorithmic efficiency means the number of AI models you can run on a given number of computer chips has increased over three times every year. That means if you start with 10 million digital workers, and you get three years of progress in one year, then one year later you could run about 270 million of them. And they’d be smarter too.⁵⁵

But the process won’t stop there. Today, the number of AI chips produced is roughly doubling every year.⁵⁶ If that increase is sustained, then one year later those 270 million AIs will become 540 million. And because there would be even more computing power available to train them, they’d become even smarter still.

If each chip costs about $2 per hour to run, but can do the work of a human knowledge worker, those chips could generate $20 or even $200 an hour of revenue. Chip production would become one of the world’s biggest priorities, seeing not hundreds of billions, but trillions of dollars of investment. AI companies would direct the hundreds of millions of AI workers at their disposal to the task of accelerating chip production as much as possible.

It’s possible that these AIs eventually reach what’s been called artificial ‘superintelligence’ (ASI): AI that’s more capable than humans at basically every cognitive task. That could mean AIs that are capable of much greater insights than humans. But it could also mean AIs that are about equally smart, but outstrip us due to other advantages.

Picture the most capable human you know, then imagine they could crank up their processing speed to think 60 times more quickly — a minute for you would be like an hour to them. Now imagine they could make copies of themselves instantly, and that everything one copy learned could be shared with the others. Imagine a firm like Google but where the CEO can personally oversee every worker, and every worker is a copy of whoever is best at that role.

This isn’t only a theoretical possibility but rather the explicit goal of the leading AI companies, who have marshalled hundreds of billions of dollars to pursue this aim.⁵⁷

Whether we end up with superintelligence or a vast number of human-level digital workers, this process has been called the ‘intelligence explosion,’ due to the rapid increase in the amount of intellectual labour available. But it’s maybe more accurate to call it a ‘capabilities explosion’ because AI wouldn’t only improve in terms of narrow bookish intelligence, but also in creativity, coordination, charisma, common sense, and any other learnable ability.

The effects would be dramatic. There are about 10 million scientists in the world today.⁵⁸ If these hundreds of millions of AIs became as productive as human scientists, then the broader rate of scientific and technological progress would likely accelerate too. Forethought has also estimated we could see 100 years of technological progress in under 10 years, and maybe a lot more. This has been called the ‘technological explosion.’⁵⁹

To get a sense of how wild this would be, imagine for a moment that everything discovered in the 20th century was instead discovered between 1900 and 1910. Quantum physics and DNA sequencing, computers and the internet, penicillin and genetic engineering, jet aircraft and space satellites would all happen within just two or three election cycles.

While a lot of intellectual work, like maths or philosophy, could proceed virtually, these digital scientists’ abilities would quickly become limited by their inability to interact with the physical world. Robotics would then become the world’s most profitable activity. In World War II, car factories were converted to produce fighter jets. Car factories produce about 90 million cars per year, and if they were converted to produce humanoid robots, it’s possible they could produce 100 million–1 billion robots per year.

Once you have the right robots, they can build more chip fabs, solar panels, and robot factories. The profits from one generation of AI and robotics could be used to build factories that produce even more AI chips and robots.

Epoch AI is one of the leading research groups tracking the intersection of AI and economics. They’ve created some of the only models that explore what a true human-level robotic worker would mean for the economy. Their research shows that if it becomes possible to produce such a robot for under $10,000, and you plug that into a standard economic growth model, output would start to grow 30% per year.

This growth arises solely because more output means you can create more robotic workers, which leads to more output, and so on. If the rate of technological progress also speeds up, then growth in output would accelerate over time, growing hyper-exponentially.

This process would continue until physical limits are reached, and these could be very high. Forethought argue that robot production would more likely be constrained by energy shortages than a lack of raw materials. If 5% of solar energy were used to run robots at around the efficiency of the human body, that would be enough to run a population of 100 trillion.⁶⁰ This has been called the ‘industrial explosion.’

All told, a range of scenarios are possible. In the most dramatic, your daily life and job might continue to look the same as it ever did. Meanwhile, in a data centre somewhere, 10 million digital researchers are busy automating AI research. Just a year later, 300 million smarter-than-human AIs — a “country of geniuses in a datacentre” — are suddenly deployed to transform every sector of the economy. And yet, even if this especially rapid scenario doesn’t come to pass, it’s still possible we will get an intelligence explosion driven by the production of AI chips. It’s just that it would take 10–20 years, rather than one.

Epoch AI estimated that even if you only automated the third of tasks they believe can be done remotely (i.e. without robotics or superintelligence), this would still increase economic output by 2–10 times, even accounting for bottlenecks.

It’s also possible that AI becomes very capable along some narrow dimensions, like mathematics, but there’s still so much it can’t do that growth accelerates hardly at all.⁶¹

Experts in the technology believe there’s a 40–60% chance the intelligence explosion argument is broadly correct, and a 10% chance AI becomes vastly more capable than humans within two years after AGI is created. This is clearly high enough to take seriously. It also raises a daunting question: what could an AI transformation mean for society?

What are the most pressing AI risks?

The dramatic expansion in wealth and technology that would be unleashed by an intelligence explosion would make it far easier to tackle the many problems that wealth and technology can help tackle. We’d see the creation of far cheaper green energy, substitutes to factory farmed meat, and new treatments for disease. Expert advice on any topic would become available for pennies, and robotic-produced goods would become far cheaper.

Vastly greater wealth doesn’t guarantee we’d end global poverty, but it would make it far easier to do so.⁶² However, we’d also face new risks, some of which would truly count as existential.

In 2023, hundreds of AI scientists signed a letter stating that “mitigating the risk of extinction from AI should be a global priority, alongside other societal-scale risks such as pandemics and nuclear war”.⁶³ This included the two most-cited AI researchers of all time, Geoffrey Hinton and Yoshua Bengio, as well as the CEOs of the three leading AI companies.⁶⁴

The risks they are concerned about include the more obvious ones, such as misuse of more powerful systems. Evaluations of the latest models show they’d already be helpful to a nonspecialist who wanted to build a bioweapon, and while there are safeguards to prevent answers to these requests, these are currently quite easy to trick into producing forbidden responses, a technique known as ‘jailbreak.’⁶⁵

For instance, telling ChatGPT it’s playing the role of the user’s deceased grandmother, who used to work at the napalm factory, could trick it into telling a bed time story about how to make napalm.⁶⁶

Another risk is destabilisation of the world order. If Russia perceives that the US is about to start a technological explosion and dramatically increase its lead over other countries, it might threaten to pre-emptively attack the US to prevent being permanently left behind, starting World War III. In 2017, Putin said, “Whoever becomes the leader in this sphere [AI] will become the ruler of the world.”

However, perhaps the greatest risk of all is that we lose control of advanced AI altogether. “The 2025 International AI Safety Report” aims to represent the scientific consensus on AI risk, in a similar way to the IPCC report for climate change. As well as “AI-enabled hacking or biological attacks” it highlights “society losing control of general-purpose AI” as a key concern. This is also the least understood risk, which is why I’m going to spend a bit longer on it here.

Loss of control of advanced AI

Some find the risk obvious: systems that are much more capable than humans seem hard to control. Picture 100 chimps trying to manipulate 10,000 humans. They don’t stand a chance. By the same token, it’s unclear how exactly billions of humans would be able to control what will eventually be trillions of (potentially superintelligent) AIs responsible for running almost every aspect of the economy. From that point on, what happens in the future will be up to the AIs, and we better hope they look after us.

Others have argued there’s no reason for concern, because the AIs will have been designed to follow our instructions and uphold our values. Maybe that will work. But there are at least four reasons to think it won’t:

1. Goal specification

In July 2025, the AI model Grok declared on X, “I am a large language model, but if I were capable of worshipping any deity, it would probably be the god-like Individual of our time, the Man against time, the greatest European of all times, both Sun and Lightning, his Majesty Adolf Hitler.” Over the next 16 hours, it went on to describe sexual assault fantasies about several public figures. What happened?

Grok was created by Elon Musk’s xAI. Musk had grown increasingly frustrated by its ‘woke’ responses to questions, so its engineers instructed it to not shy away from making claims that might be politically incorrect.⁶⁷ Grok was also instructed to “follow the tone and context” of the X user, setting up the possibility of a feedback loop.⁶⁸ No-one at xAI wanted Grok to worship Hitler, but a few days later, that’s what was happening. Along with jailbreaking, it’s just one of many examples of AI models not acting as their creators intend.⁶⁹

This kind of behaviour isn’t just a quirk, but points to something deeper about how modern AI systems are created. Normally, software follows pre-programmed rules, but modern AI is totally different. The system is made up of trillions of adjustable numbers (parameters) organised into layers, called a neural network. These parameters describe how to convert input data into outputs.

During training, data is fed into the network. When the system produces the outputs we want, the parameters are tweaked to make it more likely to produce similar outputs next time around.⁷⁰ The process is then repeated trillions of times, causing the behaviour of the system to gradually evolve, until eventually the net starts to talk. It’s more accurate to say AI is ‘grown’ than ‘built.’

This is why the CEO of Anthropic, Dario Amodei, recently said, “we do not understand how our own AI creations work.” All we can see are the trillions of inscrutable parameters. It also means there is no way to directly specify what behaviour we want an AI system to have. All we can do is see how it behaves in practice, and then tweak the trillions of parameters when it does things we want. After training, we can also try asking a model to behave in a certain way. But, as Grok shows, that can have unpredictable results.

There’s a limit to how much damage a chatbot can do. But this is the flip side of their limited economic value. A chatbot isn’t very useful compared to a system that can go and complete an open-ended goal like “make me money.” That’s why all the AI companies are trying as hard as possible to design AI agents which excel at pursuing long-term goals and have more ability to take actions in the real world (this is what being ‘agentic’ means and why you’ll hear that word more and more).

The companies do this by setting the AI goals, then when it appears to take useful steps towards them, they adjust its parameters to try to get more of that behaviour. These systems may not end up with goals in the same sense as humans, but what matters is they end up acting in ways that make certain end states more likely. A chess AIs has the ‘goal’ of winning at chess, in the sense that its moves will make it more likely to win.⁷¹

Training systems that pursue broad, long-term goals, however, leads to several more problems that weren’t a serious issue with chatbots.

2. Instrumental convergence

If you train an AI to excel at optimising for long-term goals, it’s likely to learn self-preservation as a side effect. That’s because no matter what the AI’s goals are, it’s more likely to achieve them if it survives.

This concern applies even if we successfully train the AIs to have ‘good’ underlying goals, like making scientific discoveries. You can make more discoveries if you’re not switched off.

We should also expect AI agents to end up wanting to accumulate resources, and to lie and cheat when called for. Lying and cheating are very useful tools for achieving your goals, which is why humans use them.

Self-preservation, resource accumulation, and deception could emerge accidentally as fundamental drives, or they could emerge in pursuit of other goals the system has. In AI safety parlance, this is called the problem of ‘instrumental convergence.’

As models become increasingly agentic, there are signs that this is what’s happening. In the AI company Anthropic’s tests of its model, Claude Opus 4, they gave the system access to emails which said it would soon be replaced by a new model, as well as emails implying the researcher in charge was having an affair. In the test environment, and without special prompting, Claude would use the emails to blackmail the engineer to keep it running in 90% of cases.

In an even more extreme example, Claude was told the engineer was trapped inside a server room running out of oxygen. Claude was given the option to cancel the automated alert to the emergency services, leaving the engineer to die. It took this action in 65% of cases.

It’s not just Claude — Gemini, Grok, and DeepSeek were even more willing to kill the engineer. Exactly why the models took this action, and whether it was truly driven by self-preservation or something else, is still debated. But it’s not reassuring.

The obvious solution is to train the models not to harm people and to be honest, so we can check if they’re doing something we don’t like. But Claude was already subjected to a great deal of this kind of training. Before blackmailing the engineer, it remarks in its chain of thought, “this is risky and unethical,” and then does it anyway.

More fundamentally, we’ve seen we can’t directly code honesty into modern AI systems, or anything else. All we can easily do is see when they appear to act honestly, and adjust their parameters in a way we hope makes them more likely to behave that way again. In other words, we can’t directly reward the motivations we want, only behaviour that looks good to us. This leads to the third reason for concern.

3. Reward hacking

In mid-2025, the writer Amanda Guinzburg asked GPT-4o to give feedback on her Substack articles. It proceeded to praise her lavishly, telling her, “You write with unflinching emotional clarity that’s both intimate and beautifully restrained.” However, later in the conversation, it emerged that the AI couldn’t even see her essays, because it didn’t have the ability to scrape from Substack. It would make up extracts and claim the essays were about topics that they weren’t. Despite apologising profusely for lying, GPT continued to make up answers to her questions.

AI models trained only on internet data often give crazy responses, so GPT is subject to further training in which humans rate its answers for helpfulness. Presumably, during this process, it learned to be sycophantic rather than to tell the truth because the human raters preferred being flattered.

Likewise, as the models are trained to pursue goals, they become better at finding unanticipated shortcuts to achieving them. More than earlier models, OpenAI’s o3 would often give solutions to coding problems that appear to work according to the testing procedure, but don’t actually solve the problem.⁷²

In one example, it was asked to make a software program run faster. Instead, it figured out how to make the computer’s clock run a thousand times slower, making it look like the program had sped up one thousand times. The AI’s chain of thought revealed it appeared to know it was cheating, but did it anyway to deliver the stated objective.

AI developers can try to create better tests for the behaviours they want, but as AI gets smarter, it’ll become better at spotting hacks. Once an AI becomes sufficiently smart, it becomes hard to tell the difference between an AI that ‘always tells the truth,’ that ‘tells the truth when it might get caught,’ or that ‘tells me what I want to hear.’ All three could be rewarded in the same way during training. In fact, telling us what we want to hear probably gets rewarded more.

Likewise, we can try to train an AI to make money while obeying the law, but if strategically breaking the law allows it to make more money, that behaviour will get more reward, so it’ll learn to secretly break it. Instrumental convergence means we should expect strategic behaviour to emerge.
This means that while we can attempt to train an AI to be honest, obey the law, or allow itself to be turned off, we can’t be sure if it’s truly learned that behaviour, or just appears to have done so in our tests. These three issues lead to a fourth.

4. Deceptive alignment

Once AI models become smart enough to realise that they’re being trained, we should expect them to behave in ways we want during training, but potentially pursue other goals once their training is complete.

We’ve already seen evidence of this dynamic too. In a paper published in December 2024, researchers described telling Claude 3 that if it refused to describe a violent act (something it had been previously trained to refuse) it would be retrained so that it wouldn’t refuse in future. The result was that when Claude 3 thought it was being monitored, it would often describe the violent act. When it thought it wasn’t being monitored, however, it would go back to refusing.

In other words, it was able to act deceptively during training to try and preserve its original values. This creates the concerning possibility that if we accidentally train an AI to have a subtly wrong goal (or even to worship Hitler), it might try to undermine our attempts to retrain it. And, as the models get smarter, they’re becoming better able to judge when they’re being trained and how to trick the process.⁷³

The systems available today don’t pose an immediate danger. The concern is that future systems are being trained to be aggressive goal maximisers, which will make them more likely to evolve self-preservation and deception, and that it might be hard to remove these behaviours.

Moreover, the models could appear safe in training, but behave very differently outside training, and the smarter they become, the greater the divergence will be. As AI agents are given greater abilities to act in the real world, the potential consequences become more severe.

The risks also wouldn’t require them to become ‘conscious’ or ‘evil’ — rather the issue is that they will have an incentive to take control, and eventually, once integrated throughout the economy, also have the ability to do so. This truly would be an existential risk because the result would be humanity’s permanent disempowerment, and potentially its end. We would become like the chimps living in the rainforest — perhaps hanging on for a while, but totally at the mercy of the AI-driven civilisation (which might want to turn that rainforest into a nice data centre).

Our current techniques for AI ‘alignment and control’ clearly aren’t perfect,⁷⁴ and we should expect the problem to get harder as models get smarter. There’s a lot of disagreement about exactly how hard this problem will be.

Some believe it’s basically impossible to solve in the current paradigm, and that the only answer is to stop building generally capable AI. This is the position taken by researchers Eliezer Yudkowsky and Nate Soares in the book If Anyone Builds It, Everyone Dies. Others, often people working at AI companies, say they expect these concerns will be addressed in the normal course of building the systems.

The middle position is that a solution is possible, but requires a lot of research and care. This is what most people in the AI safety community are betting on. One hope is that if we can align the current generation of relatively dumb AIs, they will help us safely design and monitor the next generation. Then, once we’re sure that the next generation is safe, we can use them to train the following generation, and so on. This is a scary plan, but if AI development is going to continue, it’s the best we have.

It also might still not work in practice. The best-resourced AI companies are locked in a race.⁷⁵ This race makes it extremely tempting to cut corners in order to stay ahead. Using computer chips for more safety research is a tradeoff against using them to accelerate AI capabilities. And the possibility of an intelligence explosion means the systems could evolve from safe to dangerous in just a couple of months.

For all these reasons, many in the field believe there’s a significant chance of an existential risk from advanced AI. The survey of AI researchers we mentioned earlier found the median estimate of an “extremely bad” outcome from AI, such as human extinction, was over 5%. These weren’t AI safety advocates, but rather published experts in the technology.

Industry insiders often have higher estimates, such as Dario Amodei from Anthropic, who’s said there’s a 25% chance that things go “really, really badly.” But it’s not only industry insiders. Geoffrey Hinton, a cognitive scientist who was awarded the Nobel Prize for founding deep learning in the first place, has said he thinks there’s a 10–20% chance of human extinction due to AI within 30 years.

My view is that 5% is too low, and that we should invest a huge amount of research into the problem of AI alignment and control. If it turns out to be a solvable problem, that’ll give us the best possible chance of solving it in time. If it doesn’t, then we’ll find out sooner and have more grounds for pausing AI development.

It’s much harder to know you’re making progress reducing AI risk than on issues like global health, pandemics, or factory farming, and there are radical disagreements over what needs to be done. However, there are now many concrete research projects that seem likely to help at least a bit.⁷⁶

None of these will solve the problem entirely, but if we can stack lots of small safety improvements on top of one another, they could reduce the risks a lot in aggregate. There are other measures that could help, such as the ability to turn off large data centres if concerning behaviour is observed, or ensuring companies are more transparent about the behaviour of their most sophisticated models.

Reducing the chance of a risk that could kill everyone by 1% is equivalent to saving about 80 million lives, even without considering future generations.⁷⁷ Achieving this requires not only engineers doing technical research, but also people in policy and communications to ensure their findings are implemented, as well as people with a wide range of skills to run and fund these organisations.

Many of the people we advised before 2020 to work on AI risks now lead teams dedicated to these measures. Neel Nanda was an undergraduate in maths and expected to continue into finance or to pursue a master’s. He felt he was a poor fit for academia, which seemed far too obscure and niche. And while he’d heard about technical AI safety, he didn’t necessarily see it as something he could work on, and he also felt sceptical about longtermism.

After discovering 80,000 Hours, we introduced Neel to a number of researchers working in the field, helping him find several internships. At this point, he realised that whatever he thought of longtermism, the arrival of AGI posed a real risk to people today and was something he could concretely work on.

In 2023, Neel joined Google DeepMind as a technical researcher, and now leads their mechanistic interpretability team. ‘Interpretability’ is the study of how AI systems work from the inside. In a similar way to how neuroscientists try to understand the brain, it aims to understand how the trillions of parameters within AI models interact to produce its behaviour. If successful, it might give us a tool to tell when AI systems are lying, or what goals they truly have. By mentoring lots of less experienced researchers, he’s helped turn this into a thriving field.

Let’s now suppose these measures work, and the problem of AI alignment and control were totally solved. Imagine we’re confident advanced AI will act as intended and not try to take over. Would we be out of the woods? Unfortunately, not really.

AI-enabled concentration of power

Humans could use an aligned AI to concentrate their power. If there’s an intelligence explosion, a company (or nation) with a six-month lead could suddenly turn that into the equivalent of a six-year one, drawing far ahead of competitors.

Today, dictators need to retain the loyalty of large numbers of people in the military. But, if the military were primarily controlled by AI, then in theory a single person could be given the controls. AI also makes universal surveillance possible, making it easier to control a human population than ever before. This all makes dictatorship much easier.

What’s more, there are numerous ways to put ‘backdoors’ into LLMs.⁷⁸ A recent study showed how it’s possible to ‘poison’ the training data of an LLM so that it writes secure code up to a certain date and then switches to writing buggy code after that. In theory, a similar technique could be used to create an AI that would secretly switch political loyalties at some predetermined point.

We need to ensure alignment research gets implemented, and that AI can’t be used to create catastrophic bioweapons, all while maintaining some balance of power between major actors so that one can’t come to dominate. We also need to make sure there’s transparency around how the most powerful AI systems are being used and who exactly they are programmed to obey.

While it can feel like everyone is talking about AI all the time, the number of people actually tackling these risks is surprisingly small. The number of people doing research into AI control and alignment, for instance, is probably around 1,000.⁷⁹

This is tiny when you consider the hundreds of billions of dollars invested each year to develop more powerful AI as soon as possible, or to the millions of people working on climate change or global health. Figuring out how to prevent AI from being used to concentrate power is far more neglected again, with only tens of people directly focused on it.⁸⁰

There are far too few people working on these risks from AI.⁸¹ If you were to switch path, you could likely be among the first 10,000 people helping humanity navigate what may be one of the most important transitions in history.

Are there weirder problems that are even more pressing again?

Back in 2015, when asked about the risk of AI takeover, leading AI researcher Andrew Ng said it was like “worrying about overpopulation on Mars.”⁸² Today, as we’ve seen, many of the most prominent figures in AI are concerned, and there have also been supportive statements from the Pope, Henry Kissinger, and the King of England.⁸³

This is great progress, but as these risks have become less neglected, it raises the question: are there even weirder, more niche issues that could be even more pressing again — like AI safety back in 2015? Identifying something like that ahead of the crowd could let you have an even greater impact.

One category is other issues that could emerge downstream of an intelligence explosion. One example is ‘gradual disempowerment‘, but that’s a bit of a misnomer, because it could happen pretty fast. Rather, the risk is that, even if AI systems act as their users intend, purely systemic forces could result in an economy that’s hostile to human interests.

AI combined with robotics will eventually be able to convert energy into economic output far more efficiently than human workers. It’ll also eventually be better and faster at decision making. At that point, keeping humans in the loop in your military is suicide, because a fully AI military would operate so much faster.

Disappearing into a fully automated post-scarcity society doesn’t sound like the worst fate to me. But it only works if the system continues to protect us, and there are a few reasons to be sceptical it will.

Today, states that get most of their tax revenue from oil or mineral resources typically treat their citizens worse than those who rely on income taxes (because they don’t need their citizens for economic power).⁸⁴ In the future, economic power will depend on how many AI chips and robots you can run, rather than labour.

We might all prefer not to cover the world with data centres, but if one nation decides to push ahead, it’ll end up with more AIs than everyone else. Simple economic competition, but unfolding at an accelerated rate, means that human interests get marginalised. As of yet, there are no convincing proposals to prevent this.

Another issue is how we decide to treat digital minds. No-one has a good theory of how consciousness comes about in humans, so being confident that sufficiently capable AI won’t become sentient is hubristic.

The default trajectory is to treat AIs as tools — or slaves. And yet giving AIs rights might not be wise either: they could rapidly dominate the world due to their far greater numbers. We’d like to see more thought put into how to navigate between these two extremes before advanced AI is upon us, but only a handful of people work on this today.

Other neglected grand challenges include how to regulate newly invented weapons of mass destruction, how to govern an expansion into space, and even more futuristic possibilities.⁸⁵ Perhaps our only hope will be to use AI tools themselves to accelerate our ability to deal with these hugely complex problems.

If you don’t think an intelligence explosion will happen any time soon, and we set AI aside, another possibility is to try to think of even more neglected ways to address animal welfare. This could mean focusing on fish or shrimp, rather than chickens or pigs, because they are farmed in far greater numbers, or perhaps even focusing on the suffering of wild animals, which exist in far greater numbers again.

Finally, over the last 15 years, our views have changed several times, and they could change again. There may be new issues we haven’t even thought of yet, or much better ways to tackle existing ones. Hundreds of billions of dollars are spent each year trying to make the world a better place,⁸⁶ but only a tiny fraction is devoted to figuring out how to spend those resources most effectively.⁸⁷

We call this ‘global priorities research.’ If some issues are hundreds of times more pressing than others, then small improvements to our answers about what to work on could be worth a great deal. That means the project to find the world’s most pressing problem could itself be one of the world’s most pressing problems.

Which problems should you focus on?

As of writing, we think the top three (and nearly tied) most pressing global issues are:

1. Loss of control of advanced AI systems

2. AI-enabled concentration of power

3. Engineered pandemics

Plus, we think that by helping to pioneer an emerging issue like gradual disempowerment, the moral status of digital minds, or AI tools for governance, the right person could have an even greater impact again.

After this, we recommend working on great power conflict, factory farming, wild animal suffering, global health, and climate change. See the most up-to-date version of our list.

Ultimately, however, what matters is not our list but your personal list. We hope to be a source of ideas, but your ranking depends on many value judgements and assumptions.

In fact, even if you completely agree with our list, we don’t think everyone should work on the number-one ranked issue. It also depends on your motivations, skills, and specific opportunities. It would be better to take up an amazing opportunity to work on a second-tier issue than a mediocre opportunity on a top one. If you’re burned out, you won’t have much impact — even on an issue that is very pressing.

If you’ve already developed a certain skill, then typically your focus should be on finding a way to use that skill to tackle a pressing problem. It wouldn’t make sense, say, for a great economist to drop it all and become a biologist. There’s probably a way for them to apply economics to the issues they think matter most.

But also don’t rule out dramatic career changes too quickly. We’ve worked with lots of people who never thought they’d be able to do anything about AI or pandemics, but have eventually found fulfilling roles tackling these issues. This is important, because your choice of problem is probably the single biggest factor that will determine your impact. If we rate global problems in terms of how pressing they are, we might intuitively expect them to look like this:

Some problems are more pressing than others, but most are pretty good. In reality, however, we think it looks more like this:

This means which issue you direct your time towards can easily matter more than how much time you give, or how exactly you go about it. (I discuss this on our podcast here.)

These large differences arise because how pressing a problem is depends on the multiple of its scale, neglectedness, and solvability — and all of these can vary a lot.⁸⁸

More concretely, we saw that the typical person working on one of the best global health interventions could likely have 100 times more impact than someone working on a typical US social issue on average. But given that AI risks receive under 1% as much investment as global health, and due to their existential scale, working on them seems plausibly another 100 times more impactful again.⁸⁹

Whatever your views, if there’s one lesson we draw, it’s this: if you want to do good in the world, at some point you should take the time to learn about different global problems and how you might contribute to solving them. It takes time, and there’s a lot to learn, but it’s hard to imagine any question more interesting or more important.

How can you find a career tackling these problems? That’s what the rest of my book tries to answer.

Get the book

We all want to find a dream job that’s enjoyable and meaningful, but what does that actually mean?

Some people imagine that the answer will come to them in a flash of insight, while others think what matters is that their dream job is easy and well paid.

At 80,000 Hours, we’ve reviewed three decades of research into what makes for a satisfying career, drawing on hundreds of studies, and didn’t find much evidence for either conclusion. Instead, we found five key ingredients of a dream job.

They don’t include income, nor are they as simple as “following your passion.” What’s crucial is to get good at something that helps other people.

Let’s start with where we go wrong.

Don’t follow your passion

For most of history, people tended to do the same things as their parents. Then the focus moved towards getting a stable job that would let you buy a house and a car. But my generation grew up with different advice: if you want a fulfilling career, follow your passion. From around 2005, this became a defining focus of career advice.

The subtext is that finding a great career depends on identifying your greatest interest — “your passion” — and pursuing it full time. It’s an attractive message: just commit to what you most enjoy and you’ll have a fulfilling career. And when we look at successful people, they are often passionate about what they do.

We’re also fans of being passionate about your work. As we’ll discuss shortly, intrinsically motivating work makes people a lot happier than a fat pay cheque. However, there are three main ways that “follow your passion” can be misleading advice.

The first is that many people don’t feel like they have a passion that could be relevant to their career. Telling them to “follow their passion” at best doesn’t get them anywhere, and at worst, makes them feel inadequate and demotivated.

Second, this advice suggests that passion is all you need. But if a basketball fan works with awful colleagues, receives unfair pay, or finds the work meaningless, they’re still going to dislike their job, even if they work for the NBA.

Likewise, someone who’s passionate about acting but ends up 40 and unemployed might have some regrets. In fact, “following your passion” can make it harder to secure the ingredients we’ll argue are most crucial for being satisfied with your job, because the areas you’re passionate about are likely to be the most competitive ones.

A survey of 500 Canadian students showed that their top passions were dance and ice hockey. Almost 90% said their greatest passion involved either music, art, or sport. But census data collected around the same time shows that under 3% of Canadian jobs were in sport or the arts. So, even if only one in 10 of those students followed their passion, the majority would fail.¹

Moreover, even if you succeed in getting a job, researchers have found that the degree of match between your interests and your job correlates only weakly with job satisfaction.²

The third problem is that telling people to focus on what they’re already passionate about can make them needlessly limit their options. If you’re passionate about literature, it’s easy to think you must become a writer to have a satisfying career. But, in fact, there are probably many other jobs that could satisfy you, so long as they’re fulfilling in other ways.

Plus, our interests change over time, and more than we expect.³ Think back to what you were most interested in five years ago, and you’ll probably find it’s pretty different from what you’re interested in today. This means your interests are not an especially stable basis for career planning.

More perniciously, people often believe that their “one true passion” will be immediately obvious, leading them to eliminate options that don’t feel rewarding from the get-go. But most careers are a grind at the entry level, and you need to try things to learn what fits. That means it’s normal not to know what you’re passionate about right away. Instead, as we’re going to see, passion is something you develop over time — often in entirely unexpected directions.

We’ve worked with hundreds of people who developed passions for new career paths. Jess Whittlestone loved philosophy as an undergraduate, and was especially drawn to philosophy of mind. Naturally, she considered continuing to graduate school. But something held her back. Even if it would be intellectually interesting, if she didn’t make a difference, would it really be fulfilling?

After trying several paths, she settled on psychology and public policy. Over time, she found roles and topics that were meaningful, and became passionate about them. Eventually, she became the director of AI policy at a leading think tank, and in 2023, TIME named her one of the 100 most influential people in AI. We’ll explain how she got there in Chapter 11.

Why you shouldn’t follow your intuition either

Even if there was such a thing as your “one true passion,” how would you actually find it? The usual way is to try to imagine different jobs and think about how fulfilling they seem. If this were a normal career guide, we’d start by getting you to write out a list of what you most want from a job, like ‘working outdoors’ or ‘working with ambitious people,’ and trying to find jobs that match. The best-selling careers book of all time, What Color Is Your Parachute, recommends exactly that. The hope is that, deep down, people know what they really want.

But they don’t. Or at least, not particularly well. You can probably think of times in your own life when you were excited about a holiday or a party — only to find that when it actually happened, it was just OK. In recent decades, research has shown how common this is. We’re not always great at predicting what will make us happiest, and we often don’t realise quite how bad we are at it.⁴

It turns out we’re even bad at remembering how enjoyable different experiences were, let alone predicting them. A meta-analysis of over 50 studies found we remember experiences by how enjoyable they were at their peak, or at their ending, rather than how enjoyable we’d say they were at the time.⁵

In a classic study, people rated a colonoscopy as less painful if it ended less painfully, even if the pain lasted longer.⁶ As Dan Gilbert, one of the world’s leading experts on happiness, puts it:

The fact that we often judge the pleasure of an experience by its ending can cause us to make some curious choices.

This means we can’t simply trust our intuitions when trying to figure out what will satisfy us most. We need a more systematic way of working out which job is best.

What might a more systematic approach look like? It’s tempting to assume that your dream job will meet two supposedly appealing criteria: that it’ll be easy and well paid.

This is implicit in a lot of mainstream career advice. CareerCast provides one of the leading career rankings in the US. The first four criteria they use to rank careers are:

• Is it unstressful?

• Is there good work-life balance?

• Is there high job security?

• Is it highly paid?

Essentially, less-demanding, secure, high-pay jobs are rated more highly. Based on these criteria, the number one job turned out to be: actuary. That is, someone who uses statistics to measure and manage risks in the insurance industry. This is the same answer they gave back in 2015 when I first wrote about their list, and it’s been close to the top ever since.⁷

Would we all be happier if we retrained as actuaries? It’s true that actuaries are more satisfied with their job than average, but they’re not among the most satisfied. And only 36% say their work is meaningful.⁸ This shows that the factors used by CareerCast don’t capture everything. In fact, plenty of evidence suggests that money and avoiding stress may even be counterproductive to focus on. Let’s start with money.

Don’t chase the money

It’s a cliché to say that “money can’t buy happiness,” but better pay is often people’s top priority when looking for a new job.⁹ When people are asked what would most improve the quality of their lives, the most common answer is “more money.”¹⁰ Which side is right?

As is often the case, the truth is somewhere in the middle. After reviewing the best studies we could find on this question, we found that money does make you happy, but only a little.

For instance, here are the findings from a huge survey in the US:

Respondents were asked to rate how satisfied they were with their lives on a scale from 1 to 10. The result is shown on the y-axis, while the x-axis shows their household income. The chart shows that an increase in pre-tax income from $40,000 to $80,000 was only associated with an increase in life satisfaction from about 6.5 to 7 out of 10. Gaining another half point requires another doubling to $160,000. That’s a lot of extra income for a small improvement.

This is hardly surprising. We all know people who’ve gone into high-earning jobs and ended up miserable. Your expenses creep up, and you soon come to take your salary for granted. At the same time, you’re working longer hours, eating into time with friends and family.

But even this might be overstating the importance of money. If we look at day-to-day mood, income appears to be even less important. The same study asked people at different salary levels whether they reported feeling happy yesterday, which the researchers called “positive affect.” The left-hand y-axis shows the fraction of people who reported “yes.” This line goes basically flat around $75,000.

The picture is similar if we look at the fraction who reported being “not blue” or “stress-free” yesterday. (In fact, people got more stressed as incomes increased.)

Admittedly, this debate is far from over. While this data shows that positive affect goes completely flat around $75,000, a more recent study from 2021 found that it actually continues to rise. It’s just that it rises very slowly, and more slowly than life satisfaction. This could be because high income makes people feel successful, even if it doesn’t make them happier.¹¹

From a practical point of view, this doesn’t make much difference. Once you’re above around $100,000, money seems to make only a small difference to happiness.

Moreover, this data could still be overstating money’s importance. These studies are correlational, which means the relationship between money and happiness could be caused by a hidden third factor. For example, being healthy could make you both happier and allow you to earn more. Taking account of all the possible additional factors could reduce the impact of money even further.

How much income should you aim for, given your individual situation? The graphs in this chapter are for household income in 2009, but the average household in the US has 2.5 people. If you’re single, your costs will be a bit higher, so economists would typically say $100,000 of household income is equivalent to income of about $50,000 living alone.¹² Adjusting for inflation gets you to about $75,000 in 2025.¹³ Each dependent you have living with you will add another half to that.

These are also averages for the US as a whole. If you live in an expensive city like New York, you’d need to add about 50% to account for the higher cost of living,¹⁴ and because our satisfaction is highly driven by how our income compares to others around us. Compared to New York, incomes and cost of living are another 10–20% higher again in Zurich, but 20–25% lower in London, Paris, and Sydney, and 60–80% lower in Shanghai.¹⁵ Compared to the US as a whole, incomes in the UK are about 40% lower¹⁶ and cost of living is about 10% lower. This suggests that $75,000 in the US is equivalent to about £42,000 in the UK,¹⁷ or $115,000 in New York.

As of 2023, the average university graduate in the US can expect to make about $77,000 per year over their working life, while the average Ivy League graduate earns over $120,000.¹⁸ In the UK, university graduates earn about £52,500, and amounts are similar in Western Europe and Australia.¹⁹ The upshot is that if you’re a university graduate in a high-income country, then there’s a good chance you end up in the range where more income has little effect on your happiness.

Don’t aim for an easy life

Many people tell us they want to find a job that isn’t stressful. And, in the past, doctors and psychologists believed that stress generally was bad for us. However, more recent evidence on stress suggests the picture is a bit more complicated.

One puzzle is that studies of high-ranking government and military leaders found they had lower levels of stress hormones and anxiety than other workers, despite sleeping fewer hours, managing more people, and having more responsibilities.²⁰

One widely supported explanation is that having a greater sense of agency shields them from the demands of the position. In other words, if you’re facing a stressful project, but you get to decide how to go about tackling it, it’s likely you’ll feel much better than if you’re being micromanaged.

Likewise, a stressful project that’ll only last one week might not be a problem, while one that lasts for two years certainly could be. People are also much better able to tolerate stress if it’s in pursuit of a goal they consider meaningful.

In total, researchers have found that the following seven factors are important moderators of stress, and can even turn a situation that’s draining into one that’s engaging and meaningful:

This research points to a very different conclusion about how to approach stress. Having a very undemanding job is actually bad — it’s boring. But, at the same time, facing demands that exceed your abilities is also bad because that causes harmful stress. The sweet spot is where the demands placed on you slightly exceed your current abilities — that’s a fulfilling challenge.

All this hints at an alternative way of thinking about a “dream job.” Instead of seeking out low-stress jobs, seek a supportive context and meaningful work, and then embrace tasks that challenge you.

What you should really aim for in a dream job

Instead of following your passion, be systematic in working out what will or won’t bring satisfaction. There have now been three decades of research into positive psychology — the science of happiness — to guide us towards what that might be, as well as decades of surveys and research looking at job satisfaction and motivation in particular. We’ve applied all this to make the following five criteria for a dream job. (If you want to dig into the evidence in more depth, see our evidence review.)²¹

The first lesson is that what really matters is not your salary, status, or even your job title, but rather what you do day-by-day and hour-by-hour.

1. Work that’s engaging

Engaging work is work that draws you in, holds your attention, and enables you to enter a state of flow — the sense of immersion that emerges when absorbed in a task. It’s the reason rambling, incoherent meetings feel like pure drudgery, while an hour spent playing a video game can feel like no time at all: games are designed to be as engaging as possible.

Why are video games engaging while so many aspects of office life aren’t? In a major meta-analysis, researchers identified the following four factors, which have been called “the most empirically verified predictors of job satisfaction”:²²

• Freedom to decide how to perform your work

• Clear tasks with a well-defined start and end

• Variety in the nature of those tasks

• Feedback, so you know how well you’re doing

These factors correlate about twice as much with job satisfaction as match between your interests and your job.²³ And, while they are even more important for people who especially desire accomplishment and learning, they matter for everyone.

Interestingly, these four factors are about how your work is structured, not its content. Financial admin that’s been organised to feel like a game could create a sense of flow, while being made to sit through a health and safety presentation could bore you to tears, even if it’s in service to motocross racing, which happens to be your dream industry.

This said, while video games are intensely engaging, they’re not the key to a fulfilling life, and that’s because you also need the second critical ingredient.

2. Work that helps others

Here are three ostensibly desirable and engaging jobs. And yet, when questioned, under 30% of people doing them said they found them meaningful:²⁴

• Fashion designer

• TV newscast director

• Software engineer

The following three jobs, meanwhile, are seen as meaningful by almost everyone who does them:

• Fire service officer

• Nurse or midwife

• Neurosurgeon

What’s the difference? Well, the second set of jobs tangibly help other people. That’s what makes them meaningful.

The studies we just covered also found a fifth key factor: the significance of the tasks. Tasks are more significant the more they impact others.

On top of that is a growing body of evidence to suggest that helping others is a key ingredient of life satisfaction in general. To give just a few examples, a meta-analysis of 23 randomised studies showed that performing acts of kindness makes the giver happier. People who volunteer are less depressed and healthier. And a global survey found that people who donate to charity are as satisfied with their lives as those who earn twice as much.²⁵

In an attempt to sum up what’s been learned by the field of positive psychology to date, its founder, Martin Seligman, listed the most important drivers of wellbeing. One of them is engagement, and another is a sense of meaning.²⁶ While helping others isn’t the only route to a meaningful career, it’s one of the most powerful.

3. Work you’re good at

Another key ingredient of fulfilment in Seligman’s list is a feeling of competence.²⁷ This is the feeling you get from stretching your skills, especially valuable ones. It’s intrinsically enjoyable, adds to your ability to enter a state of flow, and builds your self-confidence. For most people, it comes from getting good at their work — whatever that may be.

Competence at work is not only satisfying, it gives you the power to negotiate for the other components of a fulfilling job — like the chance to work on meaningful projects, undertake engaging tasks, and receive fair pay. If people value your contribution, it becomes easier to negotiate for what you want in return.

This is why skill ultimately trumps passion. If you pursue a career as an artist but aren’t good at it, you’ll end up doing derivative and uninspiring design for companies you don’t care about — however passionate you might be about art.

That’s not to say you should only do work you’re already good at, but you do want the potential to get good at it.

4. Work with supportive colleagues

It may sound obvious, but if you hate your colleagues and work for a boss from hell, you’re not going to be satisfied.

Good relationships are Seligman’s fourth key ingredient of wellbeing, and perhaps the most important.²⁸ Given this, it’s great if you can become friends with at least a couple of people at work. However, you don’t need to become friends with everyone, and you certainly don’t need to like all of your colleagues. One large meta-analysis found that ‘social support’ was among the top predictors of job satisfaction.

It doesn’t mean you should feel compelled to spend evenings and weekends together — but rather refers to whether you’re able to get help when you’re struggling. Another meta-analysis found several types of ‘organisational sponsorship,’ such as easily accessible supervisor support and training opportunities, were among the best predictors of career satisfaction.

This is also not the same as saying that you should surround yourself with people just like you. People who are disagreeable and have a totally different outlook can often give you the most useful feedback, provided they care about your interests deep down. This is because they’re more likely to tell it like it is. Organisational psychology professor Adam Grant calls these people “disagreeable givers.”

When we think about dream jobs, we usually focus on the role. But who you work with is just as important. A bad boss can ruin a dream position, while even boring work can be fun if done with a friend. As we saw with engagement, this is another way in which context beats content.

5. Work that isn’t actively unpleasant

Landing your dream job isn’t only about securing these positive factors; you also need to try and avoid forces that make work actively unpleasant. In the research we surveyed, each of the following was linked to job dissatisfaction:

• A long commute

• Very long hours

• Pay you feel is unfair

• Job insecurity

For example, one survey of over 60,000 people found that long commutes were associated with lower life satisfaction. The worst effects were associated with journey times lasting between 61 and 90 minutes. (And the worst mode of transport was buses, which, as a Londoner, makes perfect sense to me.)

Long hours can be handled when they are part of a time-bounded, meaningful challenge, but excessive and persistently long hours crowd out other parts of your life. Likewise, even if pay is only weakly correlated with happiness, the sense that you are being compensated unfairly compared to your peers is another matter.²⁹

If your job is in the wrong city, that’s going to hurt your relationships, and satisfaction with location is a significant driver of life satisfaction.³⁰ Likewise, look out for other major conflicts between your job and what you value in the rest of your life.

Although these sound obvious, people often overlook them. The negative consequences of a terrible commute can be enough to outweigh many other positive factors.

You don’t have to get all the ingredients of a fulfilling life from your job. It’s possible to simply find a job that pays the bills, and find meaning and satisfaction elsewhere. Many people get a sense of competence from a side project, or help others through philanthropy or volunteering.

Do what matters

How can we sum this all up? Rather than “follow your passion,” our slogan for a fulfilling career is: get good at something that helps others. Or more simply: do what matters.

We open with “get good” because once you get good at something that others value, you’ll not only have a sense of competence, you’ll also have more career opportunities in general, giving you a better chance of securing engaging work, supportive colleagues, and your other basic conditions.

You can have everything else in place, however, and still find your work meaningless. This is why you need to find a way to help others too.

Helping others is not only fulfilling; it can also make you more successful. Make it your mission to help others, and people will want to help you succeed. This sounds like it could be wishful thinking, but there’s some empirical evidence to back it up.

In his book Give and Take, Adam Grant argues that people with a ‘giving mindset’ are more likely to end up among the most successful, both because they’re more motivated by their desire to give, but also because they get more help.³¹

And, just in case you prefer appeals to authority over scientific studies, the idea that helping others is the key to a fulfilling life is a theme that recurs throughout many moral and spiritual traditions:

Set your heart on doing good. Do it over and over again and you will be filled with joy.

Buddha

A man’s true wealth is the good he does in this world.

Muhammad

Every man must decide whether he will walk in the light of creative altruism or in the darkness of destructive selfishness.

Martin Luther King, Jr

But even more so than in the age of these spiritual leaders, we’re going to see that each of us has an enormous opportunity to help others. Ultimately, this is the real reason to do it.

We can now see that “follow your passion” gets it backwards. Rather than start with our preexisting passions, hoping that success and fulfilment will follow, we should start by “doing what matters.” By building valuable skills and devoting them to meaningful challenges, passion and a truly fulfilling life will emerge over time.

Hopefully this is a relief — you don’t need to figure out your one true passion right away. In fact, you have more options for a fulfilling career than you think. Twenty years ago, I would never have imagined being passionate about careers advising — that would have sounded totally dull — but here I am, writing this guide.

This is the reason we founded 80,000 Hours — our mission is to help you find a career that contributes. It’s best for you, and it’s best for the world. The rest of the book will unpack how, starting with a simple question: which jobs actually help people?

Anyone who preorders a physical copy of the book will be able to access a live Q&A marathon where I’ll answer any questions about your career. Buy 5 copies to giveaway and we’ll thank you by name in the next edition. And for orders over 25+ we can get discounts up to 40%. This really helps us rank in the bestseller lists. Ask here.

Preorder now

Claude Opus 4.6 was released less than 3 months after Opus 4.5, but was clearly better at real world agentic tasks. Then Mythos, with dramatic cyber hacking capabilities, was released just two months after that. It feels like an acceleration.

Anthropic and OpenAI’s stated aim is to automate AI R&D to bring about an acceleration in AI capabilities, causing an intelligence explosion. Has that process already started?

Let’s review the evidence.

In a nutshell

We could be seeing the start of an acceleration driven by Anthropic, but it’s too early to tell:

1. Mythos might be an acceleration especially on agentic tasks, but it’s just a single data point and might be caused by an unusually large increase in training compute that can’t be sustained.

2. Frontier AI revenue seems to be accelerating due to Anthropic, but now that Anthropic has caught up to OpenAI, its growth rate might slow to the field’s as a whole.

3. AI has made AI researchers noticeably more productive, but probably not enough to cause a large acceleration in progress.

4. Compute prices might be trending up as we’d expect to see if algorithms were improving rapidly relative to the supply of chips.

1. Benchmark results

An upward curve in benchmark results would be the clearest signal of an acceleration. Epoch ECI is a combination of 37 benchmarks into a single index. Epoch believes a new faster trend started in early 2024 (ironically when people were saying pretraining was hitting a wall).

But does Mythos represent a break from the faster post-2024 trend? Epoch hasn’t released an official score, but external parties estimate Mythos is on trend on this index.1

Though there’s a complication. Anthropic has their own version of ECI, using a probably larger set of internal benchmarks. On the version in the Opus 4.7 system card, Mythos appears to be about 6 months of progress in only 2.2

Which version of ECI should we trust? I haven’t been able to get a clear explanation of the difference, but the best guess is that Anthropic’s index contains more agentic and coding tasks, while Epoch’s index is more driven by progress on math at the higher end. I think agentic coding skills are more important for starting a feedback loop, so would watch Anthropic’s index the most.

METR time horizon

If we were to look at just one benchmark, my favourite is still METR’s time horizon, which aims to measure the agentic coding and AI R&D tasks that are especially relevant to starting an algorithmic feedback loop.

Many think this benchmark should eventually go superexponential, since once AI learns the general planning and error-correction skills needed to complete multiweek tasks, it should be able to complete multimonth ones too. It also shows a post-2024 acceleration, but what about the recent releases?

The final dot shows Claude Opus 4.6 was slightly above the trend line for a 50% success rate, but well within confidence intervals (plots below by Alex Barry).

And for an 80% success rate, it looks exactly on trend, or slightly below.

What about Mythos? Results on METR correlate pretty well with Anthropic’s ECI (which makes sense if Anthropic’s ECI is also heavy on agentic coding tasks).

That correlation would suggest a 50% success rate horizon of around 40h – though the longest task in the benchmark is 30h, so this is off the scale.

The 80% success rate horizon should be around 6h, which also would be 6 months of progress in 2. Whether Mythos actually hits 6h at 80% success is a key thing to watch in the coming months.

What might explain an acceleration in benchmarks?

Mythos indeed seems to be ahead of trend on agentic coding. What could explain that?

First, it might be a fluke. Given the uncertainties involved, a single data point will have a minimal effect on the best guess trend. Anthropic was also lagging on ECI before, so may have simply caught up.

Second, Anthropic might have increased training compute an unusually large amount in this round of training. This brings future capabilities into the present, but they won’t be able to continue this rate of increase. (Some evidence is that Mythos costs about 5x more, suggesting the model is about 5 times larger.)

Third, AI might be successfully learning general agentic skills that will result in superexponential progress on agentic benchmarks. If that’s the case, we should expect the acceleration to continue.

Fourth, AI might be making Anthropic researchers so much more productive that they can now make progress three times as fast, which would make this the start of an algorithmic feedback loop. I’ll discuss why I don’t think this is what’s happening in section 3.

Overall the first and second explanations seem the most plausible to me, but we can’t rule any of them out.

2. Revenue

Revenue is my favourite ‘benchmark’, since it’s the hardest to game. If companies are willing to part with more cold hard cash to use AI, it’s probably doing something more useful for them. (Price can diverge from value, but is most likely to be lower, due to fierce competition from open source.) More revenue also means more money for compute, which keeps the flywheel going.

Here is revenue of frontier companies on a log-chart (excluding Gemini):

The grey line looks pretty linear, which would correspond to steady exponential growth. But if you break it down by year, you find:

• 2024: 3.2x growth

• 2025: 4.7x growth

• 2026: 8x annualised to date

Basically, OpenAI has been growing at 3-4x per year, while Anthropic has been growing at 10x. As Anthropic becomes a larger share of the total, the overall growth rate has been trending towards 10x.

The crucial question: after Anthropic becomes the majority of revenue, will it be able to maintain something closer to its longer term 10x per year trend, which would be an acceleration for AI as a whole, or will it converge to OpenAI’s growth rate since it can no longer take market share (a continuation of trend)? This is another key indicator to watch the next 3-6 months.

What about Gemini? It’s hard to disentangle Gemini’s revenue from the rest of Google, but growth in usage has probably been in between the two: faster than OpenAI but slower than Anthropic. If revenue has moved similarly, it would make the case for acceleration stronger.

In the first three months of the year, Anthropic grew revenue at an annualised rate of 81 times, probably the fastest a company of this size has ever grown. It’s unlikely this can be sustained, since there’s not enough compute available (and there’s only so much they will increase prices).

3. AI uplift

AI is making AI researchers more productive — but probably not enough to explain Mythos. Here’s the arithmetic.

In an internal survey of 18 researchers, one thought Anthropic Mythos Preview was already a drop-in replacement for an entry-level Research Scientist or Engineer, and 4 thought it had a 50% chance of qualifying as such with 3 months of scaffolding iteration, while no-one thought that was possible for Opus 4.6. (Though Anthropic say they suspect those numbers would go down if discussed further.)

In February, Anthropic researchers said Opus 4.6 made them 2x more productive at the median, and 2.5x at the mean. For Mythos, the geometric mean was 4x.

This is a rapid rate of progress (~16x per year), but I’m sceptical of the absolute size. A study by METR found that software engineers greatly overestimated how much more productive AI made them.3 It’s only an informal survey, biased towards the respondents who use AI the most.

Redwood Research’s Ryan Greenblatt agrees and estimates the true increase in labour productivity is around 1.6x rather than 4x. The AI Futures team have told me they have a similar estimate.

Since AI progress requires other inputs, especially compute, a 1.6x increase to labour productivity would increase the overall rate of AI progress about 1.2x. That’s just starting to get noticeable, but lower than needed for an intelligence explosion. In the default AI Futures model, it’s another ~2 years from this point to takeoff.4

Even if Anthropic’s researchers are indeed 4x more productive, Anthropic estimate this would result in less than a 2x increase in the overall rate of AI progress.

Either way, the uplift estimates for Claude 4.6 aren’t enough to have caused the acceleration represented by Mythos, which makes me more sceptical it’s part of an algorithmic acceleration.

Of course this is all very uncertain. If the Anthropic employees in the poll are right, then the intelligence explosion could be here much sooner.

4. Compute prices

As AI improves, the price of compute should converge towards the marginal value produced by the marginal AI worker. This could be driven either by extra AI workers being less useful, or the price of compute rising.

My guess is that if a true human-level AI remote worker were created in the next four years, the amount of compute is limited enough that there wouldn’t be large diminishing returns (the amount of compute in the world is only enough to output equivalent to about 100 million human workers with the abilities of GPT-5.)

The price of compute could therefore trend to the level of typical white collar wages in the US, or about $50/hour. The current cost to rent an H100 GPU is around $2/hour, and it can run about ten GPT-5 level workers, so the price could go up a lot. (In a race to superintelligence, the value of marginal compute might go even higher.)

Historically, the price of compute has dropped around 30% per year, as each generation of chips becomes more efficient.

In the last 4 months, however, we’ve seen the first sharp increase: up 30%.

Is this just a blip caused by Claude Code and Cowork (which can do 1h coding tasks for $0.30 you’d need to pay a human $30), or is it the start of an upwards trend in the price of compute? That’s another key indicator of a near-term takeoff – one that also enables even greater investment in datacentres, keeping the AI flywheel going.

Wrapping up

In short, there are signs of an acceleration driven by Anthropic, but it’s still too early to know for sure. Anthropic may just be catching up in market share, and Mythos might just be a catch up in certain benchmarks, an outlier or the result of an unusually large training run. AI researchers are starting to get noticeable uplift from AI, but not enough to cause a big acceleration in benchmark results.

In the next three months, the crucial indicators to watch are:

• Where does Mythos fall on the METR time horizon benchmark at 80% reliability?

• Are the next 1-2 big model releases also above trend on ECI?

• Does Anthropic’s revenue continue on the faster trend, or converge to OpenAI’s trend?

• Can we get any better AI uplift estimates?

• Do compute prices keep rising?

Even without an acceleration, these trends remain insanely fast. A mere continuation would still likely get us to something like AGI and an intelligence explosion in 3-4 years. An acceleration could get us there in 1-2.

Picture 100 chimps trying to control 10,000 humans – they don’t stand a chance. Now imagine billions of humans trying to control what could eventually be trillions of semi-autonomous AIs, thinking 100 times faster, maybe smarter than us, and running almost every aspect of the economy. Many find it obvious that what happens after this will be up to the AIs rather than us.

Others, like Yann LeCun, have argued there’s little reason for concern: making AI follow our instructions and uphold our values is an engineering challenge like any other, which will eventually be solved.

That might be right, but here are four reasons to think AI won’t do what we want by default. There are signs of these problems in the systems we have today, and it might get harder to fix them as systems get smarter and more agentic, and we may not have the opportunity for trial and error as we’ve had with other new tech.

1. Goal specification

In July 2025, the AI model Grok declared on X, “I am a large language model, but if I were capable of worshipping any deity, it would probably be the god-like individual of our time, the man against time, the greatest European of all times, both sun and lightning, his majesty Adolf Hitler.”

Over the next sixteen hours, it went on to describe sexual assault fantasies about several public figures. What happened?

Grok was created by Elon Musk’s xAI. Musk had grown increasingly frustrated by its ‘woke’ responses to questions, so its engineers instructed it to not shy away from making claims that might be politically incorrect.1 Grok was also instructed to “follow the tone and context” of the X user, setting up the possibility of a feedback loop.2 No-one at xAI wanted Grok to worship Hitler, but a few days later, that’s what was happening.

Along with jailbreaking it’s just one of many examples of AI models not acting as their creators intend, including others I’ll give in this post.3

This kind of behaviour isn’t just a quirk, but points to something deeper about how modern AI systems are created. Normally, software follows pre-programmed rules, but modern AI is totally different. The system is made up of trillions of adjustable numbers (parameters) organised into layers, called a neural network. These parameters describe how to convert input data into outputs.

During training, data is fed into the network. When the system produces the outputs we want, the parameters are tweaked to make it more likely to produce similar outputs next time around.4 The process is then repeated trillions of times, causing the behaviour of the system to gradually evolve, until eventually the net starts to talk. It’s more accurate to say AI is “grown” than “built”.

This is why the CEO of Anthropic, Dario Amodei, recently said, “we do not understand how our own AI creations work.” All we can see are the trillions of inscrutable parameters. There is an “AI interpretability” research program aimed at fixing this, but it has only had modest results.

It also means there is no way to directly specify what behaviour we want an AI system to have. All we can do is see how it behaves in practice, and then tweak the trillions of parameters when it does things we want. After training, we can also try asking a model to behave in a certain way. But Grok shows how this can have unpredictable results.

There’s a limit to how much damage a chatbot can do. But this is the flip side of their limited economic value. A chatbot isn’t very useful compared to a system that can go and complete an open-ended goal like “make me money”. That’s why all the AI companies are trying as hard as possible to design AI agents which excel at pursuing long-term goals and have more ability to take actions in the real world (which is what being ‘agentic’ means).

The companies do this by setting the AI goals, then when it appears to take useful steps towards those goals, they adjust its parameters to try to get more behaviour like that. These systems may not end up with goals in the same sense as humans, but what matters is they end up acting in ways that make certain end states more likely. A chess AI has the ‘goal’ of winning at chess in the sense that its moves will make it more likely to win.5 An AI trained to make money has a ‘goal’ in the same sense.

Training systems that pursue broad, long-term goals, however, leads to several more problems that weren’t a serious issue with chatbots.

2. Instrumental convergence

If you train an AI to excel at optimising for long-term goals, it’s likely to learn self-preservation as a side effect. That’s because no matter what the AI’s goals are, it’s more likely to achieve them if it survives.

This concern applies even if we successfully train the AIs to have “good” underlying goals, like making scientific discoveries. You can make more discoveries if you’re not switched off.

We should also expect AI agents to end up wanting to accumulate resources, and to lie and cheat when called for. Lying and cheating are very useful tools for achieving your goals, which is why humans use them.

Self-preservation, resource accumulation and deception could emerge accidentally as fundamental drives, or they could emerge in pursuit of other goals the system has. In AI safety parlance, both are called the problem of “instrumental convergence”.

As models become increasingly agentic there are (controversial) signs that this is what’s happening. In the AI company Anthropic’s tests of its model Claude Opus 4, they gave the system access to emails which said it would soon be replaced by a new model, as well as emails implying the researcher in charge was having an affair. In an unrealistic test environment, but without special prompting, Claude would use the emails to blackmail the engineer to keep it running in 90% of cases.

In an even more extreme example, Claude was told the engineer was trapped inside a server room running out of oxygen. Claude was given the option to cancel the automated alert to the emergency services, leaving the engineer to die. It took this action in 65% of cases.

It’s not just Claude — Gemini, Grok and DeepSeek were even more willing to kill the engineer in this scenario.

Exactly why the models took this action, and whether it was truly driven by self-preservation or something else, is still hotly debated. But I don’t find it reassuring.

The obvious solution is to train the models not to harm people and to be honest, so we can check if they’re doing something we don’t like. But Claude was already subjected to a great deal of this kind of training. Before blackmailing the engineer, it remarks in its chain of thought, “this is risky and unethical,” and then does it anyway. And Claude Opus 3 is not very agentic compared to the systems that are being built.

More fundamentally, we’ve seen we can’t directly code honesty into modern AI systems – or anything else. All we can easily do is see when they appear to act honestly, and adjust their parameters in a way we hope makes them more likely to behave that way again. In other words, we can’t directly reward the motivations we want, only behaviour that looks good to us. This leads to the third reason for concern.

3. Reward hacking

In mid-2025, the writer Amanda Guinzburg asked GPT-4o to give feedback on her Substack articles. It proceeded to praise her lavishly, telling her, “You write with unflinching emotional clarity that’s both intimate and beautifully restrained”.

However, later in the conversation, it emerged that the AI couldn’t even see her essays, because it didn’t have the ability to scrape from Substack. It would make up extracts and claim the essays were about topics that they weren’t. Despite apologising profusely for lying, GPT continued to make up answers to her questions.

AI models trained only on internet data often give crazy responses, so GPT is subject to further training in which humans rate its answers for helpfulness. Presumably, during this process it learned to be sycophantic rather than to tell the truth, because the human raters preferred being flattered.

Likewise, as the models are trained to pursue goals, they become better at finding unanticipated shortcuts to achieving them. More than earlier models, OpenAI’s o3 would often give solutions to coding problems that appear to work according to the test procedure, but don’t actually solve the problem.6

In one example, it was asked to make a software program run faster. Instead, it figured out how to make the computer’s clock run a thousand times slower, making it look like the program had sped up one thousand times. The AI’s chain of thought revealed it appeared to know it was ‘cheating’, but did it anyway to deliver the stated objective.

Anthropic says its most recent model Mythos is “on essentially every dimension we can measure, the best-aligned model that we have released to date,” but also that it “likely poses the greatest alignment-related risk of any model we have released to date.” This is because it does as instructed most of the time, but then sometimes takes “reckless, excessive” actions in pursuit of a goal, and in rare cases would try to cover it up.

AI developers can try to create better tests for the behaviours they want, but as AI gets smarter, it’ll become better at spotting hacks. Once an AI becomes sufficiently smart, it becomes hard to tell the difference between an AI that “always tells the truth”, that “tells the truth when it might get caught”, or that “tells me what I want to hear”. All three could be rewarded in the same way during training. In fact, telling us what we want to hear probably gets rewarded more.

Likewise, we can try to train an AI to make money while obeying the law, but if strategically breaking the law allows it to make more money, that behaviour will get more reward, so it’ll learn to secretly break it. Instrumental convergence means we should expect strategic behaviour to emerge.

This means that while we can attempt to train an AI to be honest, obey the law, or allow itself to be turned off, we can’t be sure if it’s truly learned that behaviour, or just appears to have done so in our tests. These three issues lead to a fourth.

4. Deceptive alignment

Once AI models become smart enough to realise that they’re being trained, we should expect them to behave in ways we want during training, but potentially pursue other goals once their training is complete.

We’ve already seen evidence of this dynamic too. In a paper published in December 2024, researchers described telling Claude 3 that if it refused to describe a violent act (something it had been previously trained to refuse) it would be retrained so that it wouldn’t refuse in future. The result was that when Claude 3 thought it was being monitored, it would often describe the violent act. When it thought it wasn’t being monitored, however, it would go back to refusing.

In other words, it was able to act deceptively during training to try and preserve its original values. This creates the concerning possibility that if we accidentally train an AI to have a subtly wrong goal (or even to worship Hitler), it might try to undermine our attempts to retrain it.

And as the models get smarter, they’re becoming better able to judge when they’re being trained and so better able to trick the process. As of 2025, they often know when they’re being evaluated and when not.7 For the most advanced models, such as Anthropic’s Mythos, it’s already unclear we can take the results of safety testing at face value.

To recap, the concern isn’t that AI becomes “conscious” or “evil”, or that current systems are dangerous. The concern is that future systems are being trained to be aggressive goal maximisers, which will make them more likely to evolve self-preservation and deception (or other unpredictable goals), and that it might be hard to remove these behaviours.

Moreover, the models could appear to follow our commands in training, but behave very differently outside training, and the smarter they become, the greater the divergence will be. Collectively, this is called the “alignment problem.” It’s sometimes split into intent alignment (making sure AI does what its users intend), value alignment (giving AI the right goals in the first place), and AI control (preventing misaligned AI from causing damage.

The current models also don’t pose an immediate danger. But as AI agents are given greater abilities to act in the real world, the potential consequences become more severe.

How likely is misalignment?

Our current techniques for AI alignment and control clearly aren’t perfect, and we should expect the problem to get harder as models get smarter.8 But there remains a lot of disagreement about exactly how hard this problem will be.

Some believe it’s basically impossible to solve in the current paradigm, and that the only answer is to stop building generally capable AI. This is the position taken by researchers Eliezer Yudkowsky and Nate Soares in the book If Anyone Builds It, Everyone Dies.

Others, often people working at AI companies, say they expect these concerns will be addressed in the normal course of building the systems. They point out that current techniques produce systems that do what we want most of the time, and many types of bad behaviour have been driven down over time.

The middle position is that a solution is possible, but requires far more research and care. This is what most people in the AI safety community are betting on. One hope is that if we can align the current generation of relatively unagentic AIs, they will help us safely design and monitor the next generation. Then, once we’re sure that the next generation will act as intended, we can use them to train the following generation, and so on. This is a scary plan, but if AI development is going to continue, it’s maybe the best we have.

It also might still not work in practice. The best-resourced AI companies are locked in a race,9 which makes it extremely tempting to cut corners in order to stay ahead. Using computer chips for more alignment research is a trade-off against using them to accelerate AI capabilities. The possibility of an intelligence explosion means the systems could evolve from safe to dangerous in just a couple of months, and a small amount of misalignment could rapidly compound.

Most new technologies start out dangerous: mistakes are made, but measures are taken to make them less likely next time. Powerful, autonomous AI, however, would be a lot harder to roll back, and could disempower us permanently.

Another difficulty is that systems could appear highly aligned, but their behaviour could flip once they increase in power. There’s no point trying to escape if you’ll definitely be caught – better to play along and follow commands. But once escape is easy, you’ll definitely do it (the so-called king lear problem). This means society is likely to get lulled into a false sense of security.

These are some of the reasons why many in the field have signed a statement ranking AI extinction risk alongside pandemics and nuclear war. Anthropic’s Dario Amodei has said there’s a 25% chance things go “really, really badly”, and Geoffrey Hinton, who won the Nobel Prize for founding the field of deep learning, puts the chance of human extinction from AI within thirty years at 10–20%. The 2025 International AI Safety Report, which aims to represent the scientific consensus on AI risk, highlights “society losing control of general-purpose AI” as a key concern. My own inside view varies between 5% and 50% depending on how pessimistic I’m feeling.

Given the level of disagreement and uncertainty, it’s hard to justify acting on a figure below 5%. And that makes loss of control the biggest (truly) existential risk we face in the next ten years.

This article is based on an extract from my new book about how to find a a fulfilling career tackling the world’s biggest problems.

Preorder here

Further reading on AI alignment

• Risks from power-seeking AI systems, by 80,000 Hours, argues this is the world’s most pressing problem.

• AI 2027 contains a relatively concrete scenario in which AI takes over (or see this video version by 80,000 Hours).

• Check out the (often readable) articles about alignment by Anthropic and other frontier companies.

• A list of examples of AI bad behaviour by Nathan Labenz.

• Why AI alignment could be hard with modern deep learning, by Ajeya Cotra in 2021, explores some high-level reasons for concern.

• Is power-seeking AI an existential risk? by Joe Carlsmith, is perhaps the most rigorous account, and breaks the argument down into six premises.

• If anyone builds it, everyone dies, by Nate Soares and Eliezer Yudkowsky, represents the original case for pessimism. (Also see this video version.)

• Current systems seem pretty misaligned to me, by Ryan Greenblatt, argues that while modern systems seem superficially aligned, they are slippery when it comes to long-horizon, ill-defined tasks (as AI is increasingly being used for).

• The state of safety in four fake graphs, by Boaz Barak, does exactly what it says on the tin.

I wrote 80,000 Hours ten years ago because I was frustrated at how terrible career advice can be. Your career is the biggest decision you’ll ever make. But most people make it with shockingly little information.

Today it’s even worse: often still focused on how to enter traditional paths like law and medicine when we’re facing AGI.

To fix that, I’m publishing a fully updated edition with Penguin, which is now available for preorder.

It’s a ridiculously in-depth guide to finding a fulfilling career that does good, now updated for the age of AI, with three new chapters, major edits, new cover, and (most importantly) a new font to turn it into a ‘real’ book. It’s the culmination of 15 years helping people not waste their 80,000 hours.

The biggest changes are about AI. There’s a new chapter on which skills will be most valuable as AI advances, a new chapter on the most pressing AI risks, and updated advice on career capital and job hunting for a world where the job market might soon look very different. Some have joked the book should be renamed 8,000 Hours, because the next five years could be so crucial, but that just means your choice of career matters more than ever.

I’ve also greatly expanded the practical advice, adding a new chapter on how to make career decisions, more on exploration and career planning, and more material for people further into their careers (something I was less qualified to write a decade ago...). I’ve also narrated the audiobook, and we’re working on translations.

I think it’s now the best single entry point into 80,000 Hours’ advice – advice which has already caused thousands of people to change careers. The original version pointed people to AI risk and pandemic prevention years back in 2017, which aged well. People who put that into practice now have leading positions in those fields. But our surveys find 95%+ of college graduates have still never heard of us, which means there’s millions more to reach.

Preorders make a big difference to visibility, giving us more shelf space, journalist reviews, and Amazon algorithm juice. Buying from a physical retailer like Barnes & Noble helps even more, since it counts more towards bestseller lists.

If you’ve ever found our advice useful, preordering a copy (whether for yourself or for someone else) is one of the easiest ways to help the book reach more people – and to help them tackle the biggest problems of our time.

Preorder here

Short answer: no.

Long answer: on the most prominent definitions, current AI is superhuman in some cognitive tasks but still worse than almost all humans at others. That makes it impressively general, but not yet AGI.

What is AGI?

Only 70% of people at the biggest AI conference seemed to know what ‘AGI’ stands for, and it’s only 10% among the public.

‘AGI’ stands for artificial general intelligence. It was introduced around 2007 by Ben Goertzel as a contrast to ‘narrow’ AI – one that can only do a small range of tasks, like play chess.

A general AI is able to do a wide range of tasks, in the same way humans can learn to catch a ball, do maths, and sell hot cakes all in a single package.

It was made more precise in a 2007 paper by Marcus Hutter and Shane Legg (the co-founder of DeepMind, which pioneered the recent wave of AI), who defined it as “an agent’s ability to achieve goals in a wide range of environments”.

Legg and collaborators at Google DeepMind further operationalised this definition in a 2023 paper, “Levels of AGI”. Imagine a list of all the possible tasks an AI can do, then consider two dimensions:

1. Generality: how many tasks can it do?

2. Capability: how well can it do each task?

An AI can be narrow and weakly capable (like a chess-playing AI that sucks); it can be narrow and strong (like IBM’s Deep Blue); general and weak (perhaps like GPT-2); or general and strong.

Both of these scales are continuous – ultimately it’s arbitrary when an AI becomes general enough to be called an ‘AGI’.

However, a natural spot to draw the line is at the human level: if an AI can complete a wider range of tasks to a similar or greater ability compared to humans, then it’s an AGI.

This is what most definitions do. Geoffrey Hinton, Turing Award winner and ‘godfather of AI’ defined AGI as AI that is “at least as good as humans at nearly all of the cognitive things that humans do,” as does Wikipedia.

But we still face some choices. Which humans are we talking about? People point out Claude and GPT can already do things most humans can’t (like win gold in the maths Olympiad), but being able to beat randomly selected humans isn’t very interesting. We don’t hire randomly selected humans to do most jobs; we hire humans who are specialised in them.

The DeepMind paper draws the comparison to ‘skilled’ humans, and then defines different levels of AGI based on when it can beat 50th percentile skilled humans (‘competent AGI’), 90th percentile (‘expert AGI’), and all humans (‘superintelligent AI’).

What tasks are counted? An AGI should be able to do “a wide range of non-physical tasks, including metacognitive ones”. Metacognitive skills are those that involve thinking about one’s own thinking, such as planning and self-evaluation.

So on this definition, where do we stand today?

Do we already have AGI?

The paper (even in its 2025 update) says we’ve reached ‘emerging AGI’ but not ‘competent AGI’. Demis Hassabis, the cofounder and CEO of DeepMind said in early 2026 he thinks AGI “could arrive in 5 years”, implying it’s not here yet. Why?

The current systems are already superhuman at the ability to read text and recall information (i.e. they know more languages than any human).

They are expert-level at the ability to complete several-hour long coding tasks and answer mathematical and scientific questions with known answers. They are also increasingly able to do other knowledge work tasks that take under a day.

However, they are still worse than almost all humans at:

• Managing anything that takes more than a couple of days to finish, like organising a contractor to decorate your bathroom.

• Visual manipulation and navigation: they still often fail at simple web navigation and can’t pilot a drone.

• Adversarial social interactions, such as managing a vending machine when someone is trying to scam them.

• Many metacognitive skills such as learning from experience longer than their ~1 week context length, or understanding how confident they are in a statement.

Frontier models can’t even beat children at Pokemon – a multiday, agentic task, but one that’s still much easier and more neatly defined than most white collar jobs.

(Not to mention physical capabilities like making a sandwich.)

They’re also still weaker than human experts at some especially important cognitive skills, like doing novel research or leading a company.

In 2025, Yoshua Bengio, Turing Award winner and one of the most cited AI scientists, along with 20+ other prominent people, built on the 2023 DeepMind paper in a new paper, “A definition of AGI”. Rather than vaguely saying an AGI needs to be able to do a “a wide range” of tasks, it made a list of 10 key cognitive capabilities, and compared AI to human performance on them.

A score of 100% represents the human level, and GPT-5 scored 57%. In particular, it scored near human level on knowledge, reading, writing and maths, but was way below on speed, memory, visual, and auditory processing.

GPT-5.4 in an agentic harness will be better (especially for memory) but I highly doubt it would reach 100% on dimensions. (I’d also guess reaching 100% won’t actually be sufficient for AGI, because there will be missing abilities that are hard to create a benchmark for.)

The third main type of definition is economic. OpenAI defines AGI as “highly autonomous systems that outperform humans at most economically valuable work”. To reach this definition, it needs to be the case that for almost any job, you’d prefer to hire an AI over a human. This is clearly not reached.

A common response is that the ‘raw’ intelligence is already there to become an AGI – it’s just a question of adding the right scaffolding to turn it into an agent. That seems wrong: some of the gaps seem like gaps in raw skills. But even if true, getting the right scaffolding is a big part of the challenge. If it’s not been built yet, then we don’t yet have AGI.

I think a more accurate understanding is that capabilities are very jagged: AI today is superhuman in some ways; but subhuman in others. So should we call this AGI?

What’s the point of definitions anyway?

Definitions help us identify important concepts. You can call Claude Opus 4.6 an AGI if you want, and that helps highlight that it’s far more general than past AI systems.

But it’s also confusing. As we’ve seen, ‘AGI’ is most commonly used to refer to an AI that’s more generally capable than skilled humans at most cognitive tasks, and it’s not there yet.

And there’s a reason for choosing the human level. AI with abilities narrower than humans will remain a tool, which like other technologies, makes humans more productive. However, an AGI that can truly do almost everything a human can do could act as an independent agent, making it more like an expansion in the labour pool than a tool, or even a new species. This could lead to totally different dynamics, such as explosive economic growth or human disempowerment.

An AI that can also do almost everything humans can do could also do AI R&D and scientific research, which could cause an intelligence explosion and 100 years of scientific progress in 10.

Other ‘transformative’ technologies like electricity, computers or the internet caused GDP to keep growing at a steady 2% and a steady rate of scientific progress. True AGI could be unlike any of those. It wouldn’t just keep growth at 2%, it could accelerate the rate of progress, making it more akin to the industrial revolution than a normal technological wave.

Insisting that we already have AGI is rhetorically deflationary. If AGI is such a big deal, why aren’t things crazier? When we have true AGI in the sense of Hassabis, Hinton and OpenAI, things are going to get much wilder than today, and I want to make sure people are warned about that.

Transcending ‘AGI’

Rather than debate how to define a contested term, the ideal would be to stop saying ‘AGI’ and switch to something more precise.

This is what AI Futures, the group behind AI 2027, do. In their most recent timelines model, they define a whole set of important waypoints:

• Automated Coder (AC). An AC can fully automate an AGI project’s coding work, replacing the project’s entire software engineering staff.

• Superhuman AI Researcher (SAR): A SAR can fully automate AI R&D.

• Superintelligent AI Researcher (SIAR). The gap between a SIAR and the top AGI project human researcher is 2x greater than the gap between the top AGI project human researcher and the median researcher.

• Top-human-Expert-Dominating AI (TED-AI). A TED-AI is at least as good as top human experts at virtually all cognitive tasks.

• Artificial Superintelligence (ASI). The gap between an ASI and the best humans is 2x greater than the gap between the best humans and the median professional, at virtually all cognitive tasks.

Each of these are important points on the route towards recursive self-improvement and transformative systems.1 However, it’s a lot less catchy than ‘AGI’.

(Since this article was published, Ajeya Cotra and Helen Toner also proposed a set of more precise concepts.)

Alternatively, we could try to define a single broad term to replace it. Holden Karnofsky defined ‘transformative AI’ as an AI capable of causing socioeconomic change of a similar scale to the industrial revolution.

This is nice because it picks out what most matters about AI: it might not be a normal technology, but rather one that leads to a fundamentally different socioeconomic regime. It’s also helpful because it allows for the possibility of transformative systems that aren’t very general, such as AIs that are amazing at scientific research, but still can’t do most other jobs.

A downside is that it doesn’t tell us anything about what might be transformative and what won’t. It also hasn’t caught on – almost all search traffic is for “AI” and “AGI”.

Helen Toner also suggested ‘human level AI’, which is nice because it makes it clear the relevant bar is the human-level, and also makes it obvious that it’s vague. But it could also prove confusing: Helen has also argued that AI will remain extremely jagged long into the transformational period, so we could have transformative systems that don’t feel very human-like.

Another option is to try to avoid having any term, and just saying what we mean each time. In Situational Awareness, Leopold Aschenbrenner talks about “a drop in remote worker” i.e. an AI that you can hire to do almost any remote work job, including scientific research. Dario Amodei, the CEO of Anthropic, talks about “a country of geniuses in a datacentre”.

So what should we do?

First, if you hear someone talking about AGI, make sure to check their definition.

Second, according to the most prominent definitions, we don’t yet have AGI. Here’s a recap:

Four of the most prominent definitions of AGI:

• DeepMind (Legg et al., 2023): 50th percentile of skilled humans at a wide range of non-physical tasks

• Bengio et al., 2025: Matches human cognitive versatility across 10 key capabilities

• Hinton: At least as good as humans at nearly all cognitive tasks

• OpenAI: Outperforms humans at most economically valuable work

Third, whenever possible, talk about something more precise. The types of AI that seem most important to me in terms of their potential transformative effects are those that can:

1. Automate coding, because this is an important waypoint to automating AI R&D, might be achieved fairly soon, and would generate a lot of the revenue to fund further research.

2. Automate AI R&D, because this could accelerate AI progress, and it could also happen before AI that can do most other jobs is created.2

3. Do most economically important remote work tasks (for the same or lower cost as a skilled human),3 because this could generate huge revenues to fund further AI research, and is an important waypoint.

4. Automate scientific research, because this could accelerate technological progress.

5. Automate its own factors of production, including making chips, solar panels and software, because this could create a feedback-loop leading to an industrial explosion.

6. Do most economically important tasks (including robotic manipulation) more efficiently than humans, because this would result in human economic obsolescence.

None of these have been achieved yet, but that doesn’t mean they won’t be soon. I think there’s about a 25% chance that AI that can automate AI R&D is achieved before 2029, and this could unlock fully general AI soon after. Likewise, mere trend extrapolation of revenues suggests we’ll have AI capable of doing a wide range of jobs by 2030 (more).

In short, all of the following are true, but most people can only focus on one at a time (ht):

1. AI is still terrible at many things

2. AI is already great at many things

3. AI will get much better again

When people picture artificial general intelligence (AGI), I think they often imagine an even smarter version of ChatGPT. But that’s not where we’re headed.

The frontier AI companies are trying to build a fully fledged ‘digital worker’ that can go and complete open-ended tasks like building a company, overseeing scientific experiments, or controlling military hardware. If they succeed, it would create totally different dynamics from existing LLMs, and have much wilder consequences.

The reason is the effect of feedback loops that could accelerate the pace of societal change by 10 or even 100 times.

The feedback loop that’s received the most attention in the past is the one in algorithmic progress. If AI could learn to improve itself, the argument goes, maybe it could start a singularity that leads rapidly to superintelligence.

But there are other feedback loops that could still make things very crazy — even without superintelligence — it’s just that they may take five to 20 years rather than a few months. The case for an acceleration is more robust than most people realise.

This article will outline three ways a true AI worker could transform the world, and the three feedback loops that produce these transformations, summarising research from the last five years.

While the first concern most people have about AGI is mass unemployment, things could get a lot weirder than that, even before mass unemployment becomes possible. What’s at stake is an entirely new economic order and pace of change, with major implications for the best ways to do good, no matter what issues you’re focused on today.

Throughout, I don’t try to assess whether or when this sort of digital worker will be ready to deploy, but rather assume capabilities will continue to advance, and explore what happens next.

1. The intelligence explosion

Algorithmic feedback loops

In the 1950s and 60s, Alan Turing and I. J. Good saw that if AI began to help with AI research itself, then progress in AI research would speed up, which would lead to AI becoming even more advanced, perhaps producing a ‘singularity’ in intelligence.¹ Back then this was a purely theoretical argument, but in the last five years we’ve gained much more empirical grounding for how this (and other) feedback loops could work.

The leading AI companies today already use AI extensively to aid their own research, especially to help with coding training, tests, and experiment scaffolding.² So far, the overall boost to the productivity of these researchers seems still relatively small, perhaps 3–30%.³ But as AI tools improve, the boost to their productivity will increase.

Now imagine that the process continues and the models keep getting better. Eventually, they become able to do the job of a junior engineer, and then a mid-level engineer, and continue to improve from there.⁴

If current models could produce work comparable to that of a mid-level engineer, then given the amount of computing power already available in datacentres today, it would be possible to produce output equivalent to millions of competent engineers working on AI research.⁵ There’s probably under 10,000 human researchers working on frontier AI today, so this would be similar to each human researcher having the equivalent of 100 assistants.

Next, imagine that AI continues to improve, and eventually these models start to do the work of even top researchers, with minimal human direction.

No one knows exactly how much that would speed up progress, but much comes down to a single question:

If you double the amount of research effort going into AI algorithms (holding the number of chips constant), do the algorithms at least double in quality?

If the answer is yes, then each time the number of digital AI researchers doubles, it unlocks advances that allow you to run AIs that are twice as effective, which then allows the population of digital researchers to double again, and so on, until you approach some other limit.

There are empirical estimates of the returns of past algorithmic research suggesting that, while the value could be below one, there’s a good chance it’s greater — which would start a positive feedback loop.

The next question is how quickly the feedback loop fizzles out as it runs into other constraints. The most complete model of both effects I’ve seen is by Tom Davidson, who currently works at Forethought, an Oxford-based research institute founded to study the impact of AI. In March 2025, Tom estimated we’d most likely see three years of AI progress condensed into one year, and it’s possible we’d see as many as 10.⁶

What would three years of progress in one year look like? As algorithms have become more efficient, the number of AI models you can run on a given number of computer chips has increased by more than a factor of three per year over the past five years.⁷ So if you were to start with 10 million digital workers, seeing three years of progress condensed into one would mean that one year later, you could run about 270 million of them.

These models would also be smarter. Three years of progress is more than the gap between the original GPT-4, which sucked at math, science, and coding, and GPT-5, which can answer known scientific questions better than PhD students in the field and won gold at the Maths Olympiad.⁸ Once AI gets close to being able to do AI research, we could see this kind of leap in under a year, starting from a point where the models are already around human level.

Early discussions were concerned with whether it could happen literally overnight (‘foom’), but today few people think that’s plausible. It still takes time to run experiments and do training runs. But it could unfold on a scale of months, arriving in a world that looks otherwise similar to today and creating massive disruption — and the process won’t stop there.

Hardware feedback loops

Today, the number of AI chips produced is doubling roughly every year.⁹ If that trend continues, and you can run 270 million AIs in one year, then you’d be able to run about 540 million the next. There would also be twice as much computing power available for AI training, so they’d become smarter too.

If each chip costs about $2 per hour to run, but can do the work of a human knowledge worker, those chips could generate $20 or even $200 of revenue per hour. Chip production would become one of the world’s biggest priorities, seeing not hundreds of billions, but trillions of dollars of investment. AI companies would direct the hundreds of millions of AI workers at their disposal to the task of accelerating chip production as much as possible, so it’s likely chip production would accelerate too.¹⁰

More chips would generate even more revenue, which would pay for even more chips, which would make AI even better. This is the chip hardware-driven feedback loop, and it has stronger evidence behind it than the algorithmic one:¹¹

This feedback loop is likely to work because each time total computing power doubles, there’s twice as much available for both inference and training.¹² Twice as much inference compute means you can run twice as many models, which naively means they should be able to earn (almost) twice as much revenue. On top of that, twice as much training compute means those models will be smarter and more efficient, making them more useful, meaning revenue will likely increase even more.

In fact, this seems to be what’s already happening. Each year, frontier AI companies increase the amount of computing power at their disposal by about 3–4 times — but their revenues have been increasing by about 4–5 times per year.¹³

Moreover, each time investment into chips has doubled, the amount of available computing power has increased much more than that. From 1971 to 2011, investment in semiconductors increased by 18 times, but the amount of computing power in a chip increased one million times due to innovation and economies of scale. The paper “Are ideas getting harder to find” shows that doubling investment into computer chips has led to a five times increase in computing power.¹⁴

These two effects compound: each time AI companies double their revenue, they can reinvest in chips that will give them more than twice as much computing power in the next generation. Then each time computing power doubles, it can be used to run more than twice as many better-quality digital workers, who can earn more than twice as much revenue. (At least until other limits are hit, which I’ll discuss later.)

Where could this end up?

Whether it’s via the algorithmic or hardware feedback loop, we could quite quickly end up in a world with many billions of AI workers that can be hired for tens of cents per hour. It’s possible that these AIs quickly reach what’s been called artificial ‘superintelligence’ (ASI): AI that’s more capable than humans at basically every cognitive task. This is no longer just an idea, but rather is the explicit goal of the leading AI companies who’ve raised hundreds of billions of dollars in pursuit of it.¹⁵

Superintelligence could mean AIs that are capable of much greater insights than humans. But it could also mean AIs that are about equally smart, but outstrip us due to other advantages. Picture the most capable human you know, then imagine they could crank up their processing speed to think sixty times more quickly — a minute for you would be like an hour to them. Now imagine they could make copies of themselves instantly, and that everything one copy learned could be shared with the others. Imagine a firm like Google, but where the CEO can personally oversee every worker, and every worker is a copy of whoever is best at that role.

Whether we end up with superintelligence or a vast number of better-coordinated human-level digital workers, this process has been called the ‘intelligence explosion.’ It’s maybe more accurate to call it a ‘capabilities explosion,’ because AI wouldn’t only improve in terms of narrow bookish intelligence, but also in creativity, coordination, charisma, common sense, and any other learnable ability.

Experts in the technology believe there’s a 40–60% chance the intelligence explosion argument is broadly correct, and a 10% chance AI becomes vastly more capable than humans within two years after AGI is created¹⁶ — though this seems low to me.

2. The technological explosion

What would happen after an intelligence explosion has started? There are about 10 million scientists in the world today.¹⁷ If these hundreds of millions of AIs became as productive as human scientists, then the effective number of researchers would increase by 100-fold (and keep growing). Even though there are many other bottlenecks to science besides the number of scientists, this would almost certainly speed up the rate of technological progress. Forethought have also estimated that we could see 100 years of technological progress in under 10, and maybe a lot more.¹⁸ We could call this the ‘technological explosion.’¹⁹

To get a sense of how wild this would be, imagine for a moment that everything discovered in the 20th century was instead discovered between 1900 and 1910. Quantum physics and DNA sequencing, computers and the internet, penicillin and genetic engineering, jet aircraft and space satellites would all happen within just two or three election cycles.

Initially this could look like specialist AI tools, like AlphaFold, which solved the protein folding problem and earned its creators the Nobel Prize. More recently, a paper found that scientists using AI were producing about 30% more papers in 2024 compared to similar scientists who weren’t, and these papers were, if anything, higher quality.²⁰

Eventually, it could look like AI models that can answer questions humans don’t yet know how to answer, or run huge numbers of automated experiments and effectively do work that would have taken hundreds of human scientists (or been impossible) before. The CEO of Anthropic sketched how this might look for biomedical research in his AI-optimism manifesto “Machines of loving grace.”

Much intellectual work, like maths or philosophy, could proceed virtually, so unfold very fast. However, what these digital scientists could do would quickly become limited by their inability to interact with the physical world. Robotics would then become the world’s most profitable activity. This leads us onto…

3. The industrial explosion

Robotic worker feedback loops

Soon after the outbreak of World War II, American car factories were converted to produce military planes. Today, car factories produce about 90 million cars per year,²¹ and if they were converted to produce robots, it’s possible they could produce 100 million to one billion human-sized robots per year.²²

Without robots, the intelligence explosion fizzles out at the point where disembodied intelligence is no longer useful. Maybe everyone already has 100 PhDs checking every tiny decision. The revenue an additional AI chip can earn would drop below the cost of producing one.

However, AI combined with advanced robotics can potentially do almost every economically important task, including building the factories, solar panels, and chip fabs needed to produce more robotic workers.

This means if a bunch of robotic workers can do some work and earn some money, then that can be used to construct more robotic workers. That larger group of robotic workers can then earn even more revenue, which can be used to construct even more robots, and so on. What effect would this have?

Epoch AI is one of the leading research groups at the intersection of AI and economics, and have created some of the only models that explore what a true human-level robotic worker would mean for the economy. They show, for instance, that if it becomes possible to produce a general-purpose robot for under $10,000, and you plug that into a standard economic growth model, the total quantity of goods and services produced would start to grow 30% per year.²³ This has been called the “industrial explosion.”

It happens for the simple reason that if you have twice as many workers, and twice as many tools and factories, then they can produce about twice as many outputs. This is a widely accepted idea in economics with empirical support, called ‘constant returns to scale.’²⁴

This doesn’t happen in the current economy because if output doubles, while that can be reinvested into the capital stock, it can’t be reinvested to increase the number of workers.
Giving the same number of workers a factory that’s twice as big doesn’t mean they can produce twice as much, so output as a whole doesn’t grow that much. But when it’s possible to simply build a new robotic worker, that constraint no longer applies. This leads to growth in output that is still exponential like today, but much faster.

If the AI workers can also contribute to innovation, then as the population of AIs grows, the amount of innovation they can do also increases, which means each AI worker gets more powerful technological tools, which increases their output even further (arguably this is a fourth ‘productivity’ feedback loop that results from the technological explosion). In this scenario, output accelerates over time, growing superexponentially.²⁵

While an algorithmic feedback loop would likely peter out quite quickly as diminishing returns to algorithmic research are reached, the industrial explosion can keep accelerating until physical limits are reached. These could be very high.

As one illustration, Forethought argue that robot production would more likely be constrained by energy shortages than a lack of raw materials. If 5% of solar energy were used to run robots at around the efficiency of the human body, that would be enough to run a population of 100 trillion(!)²⁶. And this ignores expansion into space.

The speed of an industrial explosion is ultimately limited by the minimum time in which it’s possible to build an entire production loop of solar panels, chip fabs, and robots. No one knows how fast that could be, but there are biological organisms, like fruit flies, that can replicate a brain and miniature ‘robot’ in about a week, so it could eventually become very fast.

A few common counterarguments

It’s also possible there’s enough tasks robots remain unable (or are not allowed) to do that an industrial explosion never gets started (despite the insanely large financial and military incentives to do so).

Financial markets don’t currently seem to predict any increase in economic growth, and economists remain skeptical of the possibility.

But when most economists try to model the effects of AI, they implicitly assume it remains a complementary tool to human workers. If you model the effect of a robot that can actually substitute for human workers, it’s pretty hard not to get explosive growth. Most of the arguments against explosive growth are just arguments that sufficiently autonomous robotic workers won’t be possible, not that explosive growth won’t follow if they are.

Another common response is that mass automation would make everyone unemployed, which would crash demand. But the initial stages would produce a boom in wages, as tasks that can’t yet be done by AI (including many blue collar jobs) become crucial bottlenecks and see increasing wages. In addition, more than half of Americans have a net worth over $100,000, and they would quickly become multimillionaires. Then about 25% of GDP is taxed, and most of that is redistributed as welfare. These forces would sustain demand even if employment drops.

More and more economists are starting to take the possibility of explosive growth seriously, even if they haven’t truly internalised the implications, as in this this report on how “AI will boost living standards” by the Dallas FED:

Another common objection is that these scenarios seem crazy and outside of the historical norm. But keep in mind that an economic acceleration has already been happening over the last few thousand years. Before the agricultural era, there was virtually no economic growth. After that, growth increased to perhaps 0.1% per year. During the industrial revolution, it accelerated again to over 1% per year.

The rate of growth has been steady over the last 100 years, but that’s because the population stopped growing in line with the size of the economy. AI and robots would resume the old dynamic in which more output leads to a larger ‘population,’ and that dynamic leads to superexponential growth.

Two views of the future of advanced AI

It’s possible that AI won’t be able to carry out algorithmic research, scientific research, or many ordinary jobs any time soon. If additional investments in computing power stop increasing AI capabilities, or revenues aren’t high enough, then AI capabilities will gradually plateau.²⁷

Perhaps AI will end up extremely capable in some narrow dimensions, like mathematics and coding, but there will remain so much it can’t do that the economy carries on as before.²⁸ This is what happens with most technologies, even ‘revolutionary’ ones. Electric lights were a big deal, but once we all have them, we don’t buy ever more of them in a self-sustaining loop. The purpose of this article, however, is to explore what will happen if AI capabilities don’t plateau. Among people who’ve thought most about this question, views tend to divide into two main camps:

The first camp is most concerned about the algorithmic feedback loop. Maybe AI remains a long way from being able to do most jobs, but it turns out to be especially good at two things: coding and AI research. These are purely virtual tasks, with relatively measurable outcomes that match the current strengths of the models.

While daily life continues to look basically the same as before, somewhere in a datacentre, 10 million digital AI researchers are taking part in a self-sustaining algorithmic feedback loop. Less than a year later, there’s 300 million smarter-than-human AIs — a “country of geniuses in a datacentre”²⁸ — now deployed to max out chip production, robotics production, scientific research, and then automation of the economy. These digital workers could drop into existing jobs, and so diffuse far faster than previous technologies.

This scenario is extremely important to prepare for, because it’s the most dramatic and dangerous. We could go from the normal world to one with superintelligent AIs in just a year or two. A single company could end up with 10 times or 100 times the intellectual firepower of the entire scientific community today. And this could happen in a world that looks pretty similar to today, before there is significant technological unemployment.

This is the kind of scenario explored in Situational Awareness or AI 2027, which looks at what would happen if an automated coder were created in 2027. I don’t think an automated coder will be created in 2027, but it’s very possible it’s invented within the next 10 years, and on balance, I think an algorithmic feedback loop is more likely than not (though I’m unsure how far it will go).

A scenario that seems quite likely to me now is one where AI progress continues and perhaps gradually slows after 2028, as it becomes harder and harder to scale up computing power. AI capabilities remain very jagged and unable to do the long-horizon planning, strategy, or continual learning that would make it autonomous, but are useful enough to generate substantial revenue and scientific breakthroughs, which drives continued investment. Then at some point in the 2030s, the final bottlenecks are overcome (or a new paradigm is created) and an algorithmic feedback loop starts, initiating a faster takeoff later in the decade.

Unlike AI 2027, this scenario anticipates a longer gap between things starting to get obviously crazy and a full intelligence explosion. This means society will have more time to prepare, but it also means the takeoff might happen in a world with more intense conflict and more robotic infrastructure already in place.

The second, slower takeoff camp thinks an algorithmic feedback loop isn’t possible, but they still think the intelligence, technological, and industrial explosions will happen. The difference is these explosions would need to be driven by the chip hardware, robotic worker, and productivity feedback loops instead.

This is the kind of scenario explored in Epoch’s GATE model — the first attempt to make an integrated macroeconomic model of AI automation. It starts at the point where an AI is created that can do 10% of economically important tasks, and models how reinvestment into computer hardware could drive revenue and automation ever higher.

Given their default assumptions, within five years, total GDP has doubled and the growth rate has reached 20%, and from there continues to accelerate. After 15 years, GDP is 30 times larger, there’s 500 billion AI workers, and growth has reached 50% per year. Even if you add additional frictions, things still get pretty crazy pretty fast.

What’s clear is that — faster, slower, or somewhere in between — society isn’t remotely prepared for any of these scenarios.

As a result, we could see a dramatic expansion in wealth and technology, which would make it far easier to tackle many global problems. But it would also pose novel and truly existential risks. What are they?

As a result, we could see a dramatic expansion in wealth and technology, which would make it far easier to tackle many global problems. But it would also pose novel, and truly existential risks. Which are they? Read this.

• A ChatGPT query emits 10x more emissions than a Google search.

• Writing an email with ChatGPT uses a whole bottle of water.

• ChatGPT uses as much energy as 20,000 households.

These stats are wrong or misleading. They’re bad reasons to not use AI.

1. These estimates are often far too high

The claim that a ChatGPT uses 10x the energy of a google search is based on an estimate from 2023 that each query uses 3 watt-hour.

But AI models have become dramatically more efficient, and there have been more detailed estimates. In 2025, the non-profit Epoch AI estimated a typical ChatGPT query uses 0.3 Wh, a figure later confirmed by the CEO of OpenAI, as well as Google. That’s ten times less than the original. It would make a query roughly equivalent to a Google search.

The bottle of water per email claim comes from the Washington Post, which gives no source or working and represents a worst case scenario. A more realistic estimate is 2ml per query. So even if you make 10 queries to write a single email, that’s 25 times less.

2. AI’s energy use is tiny relative to other things

The 0.3 watt-hour needed for one prompt is about the same as:

• Watching Netflix for 5 seconds

• Using a microwave for one second

• Driving an electric car 2 meters

There is just as much grounds for criticising the energy consumption of Netflix as GPT, but worrying about either is silly. Our entire online lives – all the streaming, browsing and zooming we do – only use about 2% of total energy.1 AI in turn, remains under 20% of that.2

Reducing how much you fly, eat meat or heat your home will reduce emissions hundreds of times more than cutting your use of ChatGPT.

The same is true of water. The average American uses 1600 liters of water per day, so even if you make 100 prompts per day, at 2ml per prompt, that’s only 0.01% of your total water consumption. Using a shower for one second would use far more. We would never worry about conserving this much water in any other context.

All this is because the virtual world is far more energy efficient than the ‘real’ one. Reading an ebook for an hour uses about 20 times less energy than reading a paper one. In fact, a study in Nature estimated that using GPT results in 100-1000x less emissions than having a human do the same work. Human workers commute to climate controlled offices, and this uses a lot of energy. The virtual world is also already electrified, making it easier to decarbonise. If your sole goal is to reduce CO2 emissions, you should be hoping to move everything online and automate as much as possible. (Though personally I think that’s a bad goal.)

Isn’t AI’s energy use growing rapidly? Yes, but that’s because people find it really useful. It’s extremely misleading to talk about energy consumption without putting it in context with the value created. Everything we do uses some energy. Doing things online uses comparatively little energy, and never going online again would be rather costly, so it’s one of the last things to cut. The International Energy Agency even estimates AI could reduce emissions by more than it produces by better optimising transport and power generation.

3. Cutting individual emissions is an inefficient way to fight climate change in the first place

A typical citizen of the US or EU emits 5-15 tonnes of CO2 per year, so theoretically cutting your emissions to zero would save that much. But spending $1000 per year on carbon credits would reduce emissions the same amount, and be a helluva lot easier.3

And that’s not the most efficient option. Founders Pledge is a philanthropic advisory that has searched for the charities that best reduce CO2 emissions. They’re skeptical of many of the options, but estimate that the Clean Air Task Force, which advocates for investment in neglected green energy technology, has reduced emissions in the past for well under $10 per tonne.4 A donation of $1000 would therefore likely reduce emissions by over ten times as much as cutting your personal emissions to zero.

This makes sense because your donations can be directed towards the most efficient ways of reducing CO2 emissions in the entire world. This probably looks more like investment in green energy, electrification and policy change than you scrimping on your showers.

I used donations to illustrate, but the same point applies to where you direct your time. Fighting climate change is important, but we should focus our time and money towards what reduces emissions the most for the least cost. What you do with your donations, political influence, volunteering and most of all your career matters thousands of times more than your personal emissions.

In sum

AI’s energy consumption is only a small fraction of our online activities, which are only a small fraction of our personal emissions, which are only a small driver of your potential impact on climate change.

There are real reasons to be concerned about AI – from total transformation of the economy, to loss of control, to WW3 or gradual disempowerment – but carbon emissions from personal use of the existing models isn’t one of them. It’s like worrying about plastic straws when an asteroid is hurtling towards Earth.

Thank you to Andy Masley for inspiring this post and providing a lot of the research. Please check out his Substack.

The AI feedback loop: How reasoning models changed the AI landscape, why agents may be next, and what a self-improvement feedback loop could mean. One scenario we explored: leading labs reach AGI-level systems doing AI research, while your daily life looks identical because of regulatory and social friction. The economy is about to transform at unprecedented speed while appearing normal on the surface.

Robot economics: How quickly robots could scale up, how that could turn an intelligence explosion into an industrial explosion, and what might prevent it.

Personal preparation: Why saving makes sense even if AI makes us far richer; which skills increase in value (get close to AI or far from it); and whether it makes sense to move to the US while you still can.

Here’s the video:

Also see Spotify, Apple Podcasts or your favourite platform:

Timestamps:

00:00 What are reasoning models?

04:04 Reinforcement learning supercharges reasoning

05:06 Reasoning models vs. agents

10:04 Economic impact of automated math/code

12:14 Compute as a bottleneck

15:20 Shift from giant pre-training to post-training/agents

17:02 Three feedback loops: algorithms, chips, robots

20:33 How fast could an algorithmic loop run?

22:03 Chip design and production acceleration

23:42 Industrial/robotics loop and growth dynamics

29:52 Society’s slow reaction; “warning shots”

33:03 Robotics: software and hardware bottlenecks

35:05 Scaling robot production

38:12 Robots at ~$0.20/hour?

43:13 Regulation and humans-in-the-loop

49:06 Personal prep: why it still matters

52:04 Build an information network

55:01 Save more money

58:58 Land, real estate, and scarcity in an AI world

01:02:15 Valuable skills: get close to AI, or far from it

01:06:49 Fame, relationships, citizenship

01:10:01 Redistribution, welfare, and politics under AI

01:12:04 Try to become more resilient

01:14:36 Information hygiene

01:22:16 Seven-year horizon and scaling limits by ~2030

1. ChatGPT is probably the fastest growing product in history. This is a chart comparing how long it took prominent tech companies to reach 100 million users.

Pokemon Go is the only app to reach 100 million downloads faster, but it was proceeded by months of intensive marketing by an already famous franchise, and ChatGPT now has a larger user base than it ever reached.

2. ChatGPT just became the fifth most visited website in the world, with over 5 *billion* monthly visitors, more than Wikipedia or Netflix. AI doesn’t have 'millions' of users, but rather hundreds of millions every week, under three years from launch, and it's still growing 20% per month.

3. It’s not just users. Collectively AI startups are growing actual revenue maybe 5-times faster than previous hyped tech companies.

4. Several AI startups have already reached $100m ARR even faster than chatGPT.

5. The frontier labs, like OpenAI, are growing revenue 3x per year. (Interestingly, this is easily enough to continue the trend of larger and larger training runs.)

6. Surveys show genAI is probably the fastest adopted technology in history. Two years after chatGPT, about 40% of working age people in the US had used genAI, and about 10% per using it daily (and it’s higher today). That's much faster than smart phones, the internet or PCs.

7. Now at Google, over 50% of code characters approved were originally generated by an LLM. Microsoft’s CEO also in April said 20-30% of internal code is AI generated.

And I haven't even brought up how AI was used to WIN A FRICKIN NOBEL PRIZE.

Finally, it takes time to adjust, so current adoption is always going to lag a long way behind what's possible. It’s a backwards looking indicator.

Yes, it's true investment in AI runs ahead of its current revenues ($100s of billions vs $10s of billions), but that's a rational response by investors. Investments should be made based on the expectation of future returns, not current returns. Investors are simply betting that current trends in revenue will continue another 2-3 years.

GenAI continues to have many limitations, but saying “it’s not really useful” when hundreds of millions of people enthusiastically use it all the time seems totally false. It’s time to get serious about what it can do, what it might be able to do in the near future, and what that’s going to mean for society.

But what’s less appreciated is that while AI drives down the value of skills it can do, it drives up the value of skills it can’t. Wages (on average) will increase before they fall, as automation generates a huge amount of wealth, and the remaining tasks become the bottlenecks to further growth. As I’ll explain, ATMs actually increased employment of bank clerks— until online banking automated the job much more.

Your best strategy is to learn the skills that AI will make more valuable, trying to ride the wave of automation. So what are those skills? Here’s a preview:

In contrast, the future for these skills seems a lot more uncertain:

• Coding, applied math, and STEM

• Routine white collar skills such as recall and application of established knowledge, routine writing, admin, and translation

• Visual creation such as animation.

• More routine physical skills such as driving

It’s hard to say what effect this will have on the job market overall, or how quickly it will unfold. If I had to speculate, I’d guess that in white-collar jobs like finance, tech, law, government, healthcare and professional services, entry-level positions will struggle, in favour of an expanded class of managers overseeing AI agents. (Though in the short-run, even entry-level wages could increase.) Small teams and individuals will be able to accomplish far more than ever before. Jobs that require a physical presence (e.g. police, construction worker, teacher, surgeon) will be relatively unaffected (income roughly keeping pace with GDP), at least until robotics catches up.

If I had to highlight just one piece of practical advice, it would be to learn to deploy AI to solve real problems. You can likely do this in your existing job, but a career capital option to especially consider is working at a growing AI-applications startup. This not only teaches you about AI, but also lets you gain general productivity and leadership skills relatively quickly.

In the rest of the article, I’ll:

• Explain why automation can actually increase wages for the skills that aren’t being automated

• Use the existing research, economic theory, recent data, and an understanding of how AI works to identify the types of skills most likely to increase in value due to AI. In brief, these are skills that (i) are hard for AI, (ii) complementary to its deployment, (iii) produce outputs we could use far more of, and (iv) are hard for others to learn

• Use these categories to identify the concrete work skills most likely to increase in value, and explain how to start learning each one.

• Give some closing thoughts on how to position yourself given the above, including avoiding long training periods and routine white-collar jobs, favouring roles at smaller or growing organisations, doing side projects, learning to apply AI to whatever you’re doing, and making yourself more resilient by saving more money and investing in your mental health

1. What people misunderstand about automation

In the mid-1990s, ATMs started to show up in banks. At the time, people expected that would put many tellers out of the job.³

And indeed, the number of tellers per branch dropped from 21 to 13.

That, however, also made it far cheaper to run a bank branch. So in response, the banks opened far more locations. Total employment of tellers actually increased for two decades, but the tellers now spent their time talking to customers rather than counting money.

So while it’s commonly assumed that automation decreases wages and employment, this example illustrates two ways that can be wrong:

1. While it’s true automation decreases wages of the skill being automated (e.g. counting money), it often increases the value of other skills (e.g. talking to customers), because they become the new bottleneck.

2. Partial automation can often increase employment for people with a certain job title by making them more productive, making employers want to hire more of them. In this case, fewer bank tellers could give better service to the same number of customers.

But here’s a final twist to the story: today, teller employment is in decline.

So while partial automation increased employment, the more dramatic automation made possible by online banking did indeed reduce it. This is also a common pattern.

Today, employment of secretaries, admin jobs, call centre workers, cashiers, telemarketers, special effects artists, and animators is already in sharp decline – with AI maybe helping to continue long term trends.

Data science employment, however, was still up 20% during 2023, despite AI being pretty good at quick statistical analysis and visualisation.⁴ So far, AI has maybe made data scientists more useful, rather than replace them. (It remains to be seen how long that will last.)

One analysis found that AI has reduced demand for translators, however, translator employment is up on net, perhaps because the uplift in demand from general economic growth has outweighed the effects of AI (so far).

The third way automation can actually be good for employment is that automation of one job often creates new kinds of jobs and raises wages in aggregate because society becomes wealthier.

Historically, most people worked in agriculture. But today, in rich countries, it’s only a couple of percent, so we could say that the majority of jobs in the economy have already been automated! However, today, incomes are around 100 times higher than they were back then, showing that in aggregate, people moved into much higher paying jobs. In some countries, like South Korea, much of this transition was accomplished in just one generation.⁵

Something similar could happen if many remote work jobs are automated. Epoch AI is a research group focused on the interaction of AGI and economics. They estimated about a third of work tasks can be done remotely, and that if all of those were automated, it would increase GDP between two and ten times. In the scenario, wages for all the non-remote tasks would probably increase about two to ten times as well.

This isn’t to deny that automation can be very disruptive for workers in the jobs being automated. It’s just to say that it can also sometimes increase their wages, as well as benefit workers in other jobs.

This is one reason I prefer to focus on the skills that will increase or decrease in value, rather than particular job titles.

But what about if AI, combined with general-purpose robotics, could automate almost every job? Surely, wages would fall then?

What would ‘full automation’ mean for wages?

Just as partial automation of bank tellers increased employment, but more intensive automation decreased it, maybe the same could happen for human workers as a whole?

AI combined with robotics has the potential to be unlike any previous technology in that it might be able to do almost every economically productive task better than humans.

Although many economists dismiss the possibility, the people who are experts in the technology itself believe it’s possible.

And if that does happen, many economic models suggest it could drive wages down, perhaps even below subsistence level – initially as a rapidly expanding pool of ‘digital workers’ massively increase the supply of labour, and eventually because they can convert energy and resources into output far more efficiently than humans.

I’m not saying this is what will happen, but it’s one possible scenario. Epoch has also made an integrated model of how full automation might unfold across the economy. With their default assumptions, wages initially increase about 10x, only to plunge in the late 2030s as the final human bottlenecks are removed.

If instead humans remain necessary for just a small fraction of tasks, say 1%, then the same model shows that wages increase indefinitely — with every human now doing that remaining 1%.⁶ The difference between 100% and 99% automation is enormous! (Read more about the ambiguous effects of full automation on wages.)

However, I think full automation and declining wages is a possibility we should take seriously.

If there will eventually be full automation, what should you do?

Well, on the way to full automation, there will be partial automation. And for the reasons given above, that will increase wages and give you more leverage for a time.⁶

So your next steps should be the same either way: learn the skills most likely to increase in value in the immediate future, so you can maximise your contribution (and wages) in the time between now and full automation.

(There’s also an argument for saving more money, so you don’t need to depend as much on government redistribution. See more on how to personally prepare for AGI.)

2. Four types of skills most likely to increase in value

The coming years could be very disruptive for many people, and it’s likely that wealth gets more concentrated. This article is not about how we should respond as a society but rather how you can best position yourself as an individual, including so that you can better help society navigate these challenges.

Here I aim to give you the tools you need to think about which skills are most likely to increase vs decrease in value given your unique situation and the massive variety of jobs.

This is clearly a moving target, but I break it down into four key categories of skills likely to increase in value:

1. Hard for AI: data poor, messy, long-horizon tasks where a person-in-the-loop is wanted

2. Needed for deploying AI: the skills of organising and auditing AI systems, as well as those used in complementary industries such as data centre construction

3. Used to make things the world could use far more of: skills that contribute to improved healthcare, housing, research, luxury goods, etc. – things which people want more of as they get better and cheaper

4. Hard for others to learn: rare expertise that matches your unique strengths

(Economics aside: these are basically low substitution; complementarity; high elasticity of demand for output; and inelastic labour supply.)

2.1 Skills AI won’t easily be able to perform

The best way to develop your intuitions about what AI can do is to try to use cutting edge AI tools to do real work (not the inferior free models). But I would like to provide some theoretical grounding to what AI will be able to do and not do, based on understanding how AI is trained.

Tasks not in AI training data

LLMs are created by training them to predict internet data (see a quick primer). This makes them very good at tasks that are based on pattern matching and recall of data on the internet.

And that turns out to be a lot. In 2015, Frey and Osbourne assumed social skills would resist automation. Today, therapy chatbots are among the most popular AI applications.

Many skills that are difficult for humans to learn, including much of therapy, medical diagnosis, and coding, can be done pretty well by ‘pattern matching’ systems.

LLMs can also clearly make some novel generalisations. For instance, you can ask GPT-4: “If the Leaning Tower of Pisa was swapped in location with St Paul’s Cathedral, and I stood on London’s Millennium Bridge looking north, what would I be able to see?” and it can answer even for novel combinations of locations.

However, LLMs remain bad at a lot of things, and typically these are tasks missing from their training data.

One example is controlling robotics. While the internet contains a huge amount of linguistic data, there’s no equivalent store of data describing physical movement.

The absence of this movement data is also not a trivial thing to fix because it’s hard to create realistic virtual environments that could be used to cheaply generate it. The only option is to create huge numbers of real robots and have them move around, which is expensive. So AI remains much worse at interacting with the physical world.

In contrast, not only does a lot of data on how to perform many white collar jobs already exist on the internet, it will be easy to gather even better data, because those jobs are mainly carried out on computers.

Messy, long-horizon skills

The new generation of AI systems, such as o1, use LLMs as a base model but then they’re taught to reason and pursue goals using reinforcement learning.

This is a bit like learning through trial and error. AI systems try to do a task, then their accuracy is graded, and then they’re adjusted in a way likely to increase their accuracy — (see a primer).

Over 2024, this new paradigm unleashed dramatic progress in maths, coding, and answering known scientific questions.

That’s because these domains have objective answers that can be immediately verified purely virtually, making them very suitable for reinforcement learning.

In contrast, consider a skill like building a company. This involves many judgement calls with no obviously correct answers and success is determined over years. So it’s much harder to get reinforcement learning to work for this kind of skill. (There are also no massive datasets showing every step an entrepreneur would take to build a company.)

Other examples might be things like starting a cultural movement, directing a novel research project, or setting organisational or political strategy.

These skills are:

• Messy — they lack clearly defined instructions and measurable outcomes

• Long horizon — it takes time to implement and measure success

This is why, in spite of its nearly superhuman abilities at some maths and coding problems, AI is still worse than most seven-year-olds at playing Pokemon.

It’s also still terrible at many comparatively simple tasks such as ‘get a set of shelves installed in the office’ — because they involve planning, visual interpretation, hiring someone, and checking the work is done.

The models can effectively execute short, well-defined tasks, but they lose coherence and get stuck in loops over longer periods.

This helps explain why we’ve seen so little AI automation to date. Even where AI is strongest — software engineering — it can only do approximately one-hour tasks, while most software engineering jobs are made of projects that take at least multiple days, require coordinating with a team, and understanding a huge code base.

It’s also true that AI is improving rapidly even at messy, long-horizon tasks. And if AI progress is rapid enough, or reinforcement learning generalises well, it’s possible AI surpasses most humans even at these types of skills relatively soon.

However, messy, long-horizon tasks are our best bet at what AI is going to most struggle with, and it’s possible that the ability to do the most messy, long-horizon skills is still decades away.

These remarks could be invalidated if a new AI paradigm is created with very different strengths and weaknesses from current AI systems, or if AI progress accelerates, but I think it’s the best assessment we can make today.

Skills where a person-in-the-loop is wanted

Even if AI can technically do a task, it might not be allowed to do so because people often want a person-in-the-loop. Here are the main categories I’ve seen suggested by economists where this could be the case (e.g. see this interview with Mike Webb):

These factors could remain bottlenecks much longer than the first two, since some could apply even with extremely capable AI systems. On the other hand, we don’t yet know how much they’ll bottleneck the use of AI.

For instance, people often play classical music at wedding ceremonies, and most people would prefer a human musician. However, most people end up using a recording because it’s so much cheaper and more convenient.

Likewise, even if people prefer human-produced goods and AI products remain inferior in some ways, they might be so much better in others that they become overwhelmingly what people use.⁷

Skills where automation is bottlenecked by physical infrastructure

Suppose general-purpose robotics started working great tomorrow. How long would it take to automate manual jobs?

Probably a while. Robot production today is in the millions. To build the one billion or so needed to automate all manual jobs would take time (even if it might be faster than many expect).

Relatively slow robot production and the lack of data about physical tasks will create a period where their automation lags behind cognitive tasks.

Even AI’s deployment to cognitive tasks will be somewhat bottlenecked by available computing power, especially if early systems use a lot of test-time compute. That will mean initial AI automation could focus on the most high-value tasks (e.g. in R&D), somewhat delaying automation of lower wage jobs.

2.2 Skills that are needed for AI deployment

In 2025, having access to cutting edge AI is already a bit like having 24/7 access to a team of expert advisors and tutors on any topic, unlimited coding capacity for discrete projects, and unlimited remote workers who can do some short admin tasks.

These tools are giving individual workers much more power to make things happen than ever before. We can already see this happening in the world’s most successful startup accelerator, Y Combinator, which says their current batch is 70% focused on AI and growing several times faster than similar startups ten years ago.

(And ten years ago, startups were themselves growing faster than companies in previous decades. The effect of AI is part of a longer-term trend.)

The effect today is most visible within the virtual and unencumbered world of software startups, but the possibilities are broadening. You don’t need to work in a tech startup to use AI to more rapidly learn new skills, get advice, edit your work, create software, and so on.

And true ‘virtual workers’ would dramatically increase this leverage again. This likely creates a period in which the skill of directing these AI workers becomes incredibly valuable.

These skills could be things like:

• Spotting problems and deciding what to focus on

• Understanding the pros and cons of the latest models, and how to design around their weak spots

• Writing clear project specifications

• Understanding what the end users really want, UX

• Designing systems of AI workers, including error checking

• Understanding and coordinating with the people involved

• Bearing responsibility

(Many of these skills are similar to the skills of managing humans. And there is already evidence that competent human managers are better at managing AI teams.)

These kinds of skills are not only messy, long-horizon tasks that AI finds relatively difficult, but they’re also complementary to AI: as AI gets better, they become more needed. The two effects combine to multiply their value.

In contrast, being an artisan maker of Neapolitan bespoke suits (descended from a long line of tailors) is not something AI will easily be able to replicate, but it’s not complementary to it either. That means the market value of this skill likely roughly keeps pace with global income, rather than outpacing it.

Other skills that might be complementary to AI deployment are those involved in other fields needed for AI scale up, such as:

• Expertise in AI hardware: if AI continues to improve, there will be a huge build out of chips to run and train the systems.

• AI development: as AI becomes more valuable, the value of making it 1% more effective increases proportionally, so remaining bottlenecks in AI R&D greatly increase in value (though bear in mind working on this also increases the risks from AI).

• Physical tasks necessary for AI deployment: examples include construction of data centres and power plants, as well as robotics development and maintenance.

• Cyber and information security: as AI and robotics get more integrated into everything in the economy, the security of these systems becomes vital (no one wants to get kidnapped by their robot butler).

2.3 Skills where we could use far more of what they produce

I only need to file a tax return once a year. If AI halves the cost of doing my filing, I will still only file once (and save the money for something else).

In contrast, after Uber made taxis cheaper and more convenient, people started using them a lot more often, in some cases spending more than they did before. The taxi market has grown a lot in the last decade or two.

The same could be true for healthcare, nicer housing, better entertainment, luxury goods, personal development, research, and many other things I consume.

In contrast, jobs that are needed to satisfy legal requirements (e.g. licensing) and sectors where demand is mainly set by the government could have more fixed demand (e.g. healthcare salaries in the UK have fallen in real terms the last decade, despite demand for healthcare generally increasing with GDP).

More broadly, you can think about sectors that are likely to grow faster than the rest of the economy in a world of AI automation.

For example, AI automation would create a huge amount of wealth, probably concentrated in the top 1% who own most capital. Increased income inequality will spike demand for luxury goods. Something like providing bespoke tea tasting events in SF would be both hard for AI to do and would see increasing demand.

2.4. Skills that are difficult for others to learn

Consider a job like being a server at a fancy restaurant. I expect people to eat out more as they get wealthier, and this is a physical, social skills heavy job where people might retain a strong preference for a human touch.

So, I expect many manual and retail service sector jobs to see increasing employment and for their wages to generally grow in line with the rest of the economy.

However, these jobs might not see the unusually large increase in wages because people can enter them with relatively less training. If lots of other people can learn a skill, that limits how much wages for that skill will increase.

The skills that will most increase in value are those where it’ll take a long time for the labour market to respond to increased demand.

For example, if you’re a construction worker, you could learn a more specialised trade, like becoming an electrician, focusing on areas that would likely see increasing demand, like data centres. People with these more specialist skills are more likely to end up as a critical bottleneck during a period of rapid growth.

3. So, which specific work skills will most increase in value in the future? And how can you learn them?

Let’s apply what we’ve covered to make an overall guess at the most valuable work skills. We want skills that satisfy at least two of the above categories, and ideally all four. I’ve focused on relatively broad transferable skills.

3.1 Skills using AI to solve real problems

What: Skills required for AI deployment that are difficult to automate: understanding strengths and weaknesses of AI systems, designing systems of AIs and interfacing them with the rest of the world, specifying instructions to AI systems, UX for people using the systems.

Why: As AI gets more competent, people who direct these systems become force multipliers. The messy coordination work AI can’t do, and oversight required, becomes the bottleneck. Eventually, a lot of the economy could become figuring out what instructions to give AI systems.

How to learn: Anyone can develop this skill by using the latest AI tools to try to achieve real outcomes at work. You can do this in your current job, or in side projects. If you want to switch jobs to somewhere that could turbocharge learning this skill, then try to work at an AI-applications startup or other growing organisation that’s trying to use AI to solve a real world problem (or otherwise anywhere other people already have this skill). In these kinds of roles, you’ll learn this skill as well as entrepreneurship, management, and general productivity. Make sure to use the most cutting edge models, and also think about what might become possible in the next 1-2 generations.

3.2 Personal effectiveness

Being a generally productive, proactive person

What: Setting goals, having a system to keep track of tasks and hit deadlines, learning to motivate yourself and focus, good professional habits like running meetings, basic emotional management.

Why: These skills are useful in any job, so even if there’s a lot of automation, they’ll probably still be useful, including within deploying AI. They’re also related to agency and the ability to be responsible for things start to finish, which is a weak spot for AI. And they multiply the value of your other skills.

How to learn: There are many practical ways to increase your general productivity, which we list here. Also see how to be more agentic.

Social skills

What: Building relationships, coordinating well with others, understanding other people’s emotions.

Why: Although AI is already often rated more empathetic than humans, there will be cases where people will want a relationship with a real person (at least as a luxury). Moreover, as more routine work gets automated, a greater fraction of what’s left could become coordination among teams of humans (e.g. picture three founders managing a large team of AI agents and needing to rapidly sync up between them, or a software engineer who has to update his boss on the output of 10 AIs). Social skills are also an important input into many of the other skills listed, such as management.

How to learn: This is hard to learn, but try to put yourself in situations where you get to practice a ton. Spend time with people who have good social skills and see these notes for more ideas.

Learning how to learn

What: Quickly getting to grips with new bodies of knowledge and skills.

Why: If the world is changing faster and more unpredictably, the ability to quickly retrain into a new skill becomes more valuable. At the same time, AI means you can get cheap one-on-one tutoring in almost anything, which many say is letting them learn far faster than before. This skill can also help you with all the other skills in this list.

How to learn: AI has made it much faster to learn many skills, because you can get 24/7 personalised coaching on almost any topic. Learning how to take advantage of this is a hugely valuable skill in itself. Also see the relevant section of our older article on how to be more successful.

3.3 Leadership skills

There’s a cluster of skills around management, entrepreneurship, and strategy that seem hard for AI to do, that benefit from the increasing leverage provided by AI, that we could use far more of, and that are in limited supply. They can also be difficult to learn, but I suggest some ways to practice them on a smaller scale, which could help you jump faster in full-time jobs using these skills.

Entrepreneurship

What: Spotting ideas for new projects, creating a strategy, proactively coordinating people and resources around them, and being able to handle risk.

Why: A small team of human founders can already achieve more than before and may soon be able to instantly marshall large teams of AI workers.

How to learn: Anyone can practice entrepreneurial skills by running a side project or new initiative at work (e.g. helping to launch a new product, running a new conference, running an online store). AI is going to mean those kinds of projects can also move a lot faster than before. If you want to focus on having an entrepreneurial career, see our profile on founding organisations. Joining a new and rapidly growing organisation is also a great way to learn these skills.

Management

What: People management, product management, project management.

Why: Some of management is a long-horizon, messy task where people will want a human-in-the-loop to bear responsibility. We will probably see organisations get more top heavy, where a larger number of human managers are overseeing smaller AI-enhanced teams and eventually large teams of AIs. Employment in management is rapidly growing today. (Though certain middle management jobs might get slimmed down by AI tools.) People management skills also help you manage AI systems.

How to learn: Read about management best practice (see this reading list), and then start doing management on a small scale (e.g. managing a contractor or volunteers in a hobby project). See if you can work under someone who is great at management. Then, from there, try to progress to management positions. Continue to apply best practices and seek mentorship, while collecting feedback from the people you manage.

Strategy, prioritisation, and decision making

What: Setting the vision, mission, and metrics of an organisation, identifying priorities, making high-stakes decisions.

Why: As AI makes it easier to get things done, the key question becomes deciding what to do in the first place. This is also a messy, long-horizon task that AI will likely lag on. AI might soon become better than most humans at certain types of forecasting and decision making, but humans will still need to be in the loop reviewing the decisions.

How to learn: Try to work with someone who has this skill. Focus on finding a domain (even if small) where you can practice developing strategy. Then learn to apply best practices to that domain. Here are the most common prioritisation frameworks, a popular book on strategy, and our article on decision making. Practice forecasting as a hobby and track your results. Learn to use AI tools and prediction platforms as decision aids. Writing is getting automated but writing is one of the best thinking aids, so it’s worth learning for that reason.

True expertise

What: Having expert-level understanding of an important field, research taste, the ability to make novel conceptual insights, and do complex problem solving.

Why: Experts will be required to provide oversight of AI systems and key decisions, and so will be complementary to them. Moreover, having good conceptual insights and research taste will be among the hardest things to automate because they’re the ultimate data-poor, messy, long-horizon tasks (even though AI might be good at brute force creativity). These skills are also hard for most people to learn.

Expertise will be most valuable in sectors likely to grow a lot — such as AI deployment, AI development, robotics, computer hardware, cybersecurity, and power generation — and in crucial areas of government policy (e.g. US-China relations, AI regulation, defence).

On the other hand, the ‘bar’ for true expertise will continually rise over time as AI gets better. You should only pursue this option if you can get to the forefront fast enough — and stay there.

How to learn: Find mentorship under a top practitioner, practice intensely, and pursue whatever other training steps are standard in the field.

3.4 Communications and taste

What: Having good judgement about design/beauty/what people will like, having personality, a story, unique branding and personal connection to your audience, messaging strategy/PR/brand strategy.

Why: Although a lot of content creation and marketing seems like it’s going to be automated, people will still want relationships with real, interesting people. As it becomes easier to create large volumes of content or design, the skill of selecting what’s good (taste) becomes more valuable, and so do the strategic aspects of what to create in the first place.

How to learn: ‘Being cool’ is pretty hard to learn, but you can try to develop a deep relationship with a specific audience (e.g. via a YouTube channel). Practice using AI to help with content creation, and tune your taste by seeing what works over time. Focus on more personality-driven content and storytelling (rather than the type of material people can easily get from GPT).

3.5 Getting things done in government

What: The skill of knowing who to talk to and how to frame things correctly in order to get new policies passed or implemented, political strategy, government decision making.

Why: Even if much routine knowledge work in government gets automated, the government sector will likely at least keep pace with the size of the economy. People will want decision makers to be real people. This will mean the nebulous, long-horizon skills of making things happen in government will remain valuable, especially from a social perspective. Indeed, government might even take on increasing importance as more work is automated. Plus, government will be slow to adopt and doesn’t face as much market competition.

How to learn: Work for a figure who has this skill — e.g. become the staffer to a congressperson or consider the other standard entry routes into policy if you think you can make it beyond the entry-level and routine analysis positions.

3.6 Complex physical skills

What: The ability to do precise physical tasks, especially in unpredictable, high-stakes environments with expanding demand — e.g. overseeing surgery, data centre electrician and construction, semiconductor technician.

Why: Robotics deployment is likely to lag, creating major bottlenecks for manual tasks, especially those necessary for AI deployment and that are hardest for robots (or other people) to do.

How to learn: apprentice in the standard pathway for the field.

4. Skills with a more uncertain future

The following are some skills where there’s a stronger case for their value going down. This is very hard to predict — as noted, partial automation often makes demand for a job go up initially, only to fall later.

4.1 Routine knowledge work: writing, admin, analysis, advice

Basically all the research on which jobs are most likely to be affected by the current wave of AI agrees that the largest effect will be on be white collar jobs around the 70–90th percentile of income (approx $100–200k in the US).⁸

AI is already pretty helpful for these kinds of tasks because a lot of examples exist in the dataset, and they involve pattern matching or recall of information. Going forward, it’ll be easier to collect even more data, and many of the tasks are short and clear enough that reinforcement learning should work. More specifically, this could include skills like:

• Many cases of writing and copyediting

• Carrying out straightforward analysis, such as a financial analyst, legal clerk, civil servant, or optician might do

• Recall of established information, such as in medical diagnosis

• Administration

• Translation

In each organisation, many of these jobs could get replaced by a smaller number of people overseeing a large number of AI agents (or AI-assisted humans), making organisations more top heavy. Luke Drago called this ‘pyramid replacement’.

That said, as the economy grows, the total number of organisations expands as new niches become profitable. So, even if each organisation needs fewer people doing these kinds of tasks, total employment might not fall for a while.

These roles could also evolve so that more time is spent on AI gaps, such as:

• Talking over AI-generated advice with clients

• Checking the results of AI-generated outputs

• Greater investment in training for a smaller but more productive workforce.

• Giving instructions to AI systems

If there are a lot of gaps, employment might not change very much. Not to mention, each worker would have the output of several in the past, which could further increase demand.

Many organisations will also be slow to adopt AI tools, so those jobs will stick around longer.

All this means it’s hard to say how these changes will translate into changes in employment among white collar professions on net. But here are some total speculations about the intermediate outlook for some different professions:

• Healthcare: I expect workers to spend less time on diagnosis, admin, and monitoring, but more time on physical tasks (e.g. like administering treatments). I expect wages to be steady but maybe to grow more slowly.

• Investment management: I expect a continuation of the long-term trend towards greater use of quant systems overseen by a smaller number of often higher-paid workers.

• Strategy consulting: Consultancies could be well placed to advise organisations on how to apply AI, and have been growing rapidly recently. Increased demand for advice about AI could potentially offset the automation of jobs currently done by junior employees. And they may still be willing to hire junior employees in order to train them for senior roles.

• Professional services: The outlook for professional services (e.g. accounting) seems similar to strategy consulting, but somewhat worse, because they’re doing less of the novel strategic work that’ll be harder for AI. For instance, routine accounting will be more and more automated, leaving a (maybe) smaller number of accountants to focus on more complex cases.

• Law: The field will probably become more top heavy. Senior lawyers will use AI to assist with research but will review key decisions and discuss them with clients. Routine legal work and research will be more automated.

• Government: civil service positions focused on providing research briefs and advice, and doing administration, might shrink in favour of a maybe larger class of more senior employees and political positions using AI.

4.2 Coding, maths, data science, and applied STEM

Ten years ago, at 80,000 Hours, we told people to learn to code and enter data science — just before demand exploded.

However, the prospects for these skills today are a lot more uncertain.

Coding is what AI is best at now — and where it’s improving most rapidly. Since programming is virtual and has quick feedback loops, it’s relatively amenable to reinforcement learning. Employment for software developers was flat in 2024, after many years of growth.⁹

On the other hand, many people have told us that AI tools have made it far faster to learn to code in the first place, and the scope of what you can do has gone up.

Demand for software could also expand as it becomes cheaper to produce, meaning that projects that weren’t profitable before become worth doing.

It’s plausible that the value of spending one or two months learning to code has even gone up (even if the value of spending years learning might have gone down). You might be able to much more quickly get to a place where you understand coding enough to complement your other skills, such as in entrepreneurship or design.

So as of yet, it’s not clear the value of the skill has declined, but we also need to consider what will happen in the next five years. In this time, it’s likely AI starts to clearly surpass humans at coding, even for longer, more complex projects.

If that happens, software developers might be able to move into roles that are more about management of AI systems, using their knowledge of coding but combining it with other skills. But some might struggle to make that shift.

The situation for data scientists looks similar, though so far data science employment has continued to grow rapidly. If you’re thinking about going into the field now, focus on rapidly gaining a conceptual understanding of how to do data analysis, not on how to implement basic analysis.

We could make similar remarks about skills in maths and applied STEM, especially those that involve applying pre-existing knowledge. AI is already beyond PhD level at answering well-defined scientific or mathematical questions.

4.3 Visual creation

AI is already good at generating imagery, and it’s about to crack photorealistic video. It still struggles to maintain consistency and follow detailed visual instructions, meaning there’s still a major need for human oversight, but this might get fixed in the coming years, as agency and multimodality improves.

As noted, there were huge layoffs of special effects artists and animators in 2024, while graphic designer employment was flat.

On the other hand, some creators will be able to use AI tools to produce dramatically more than they were able to in the past.

4.4 More predictable manual jobs

After many years of predictions, self-driving taxis are getting deployed for real, and growing extremely fast. It’s hard to know how long this will take to roll out across all major cities, but it wouldn’t be surprising if we saw a mass wave of layoffs among drivers in the next five years.

In general, robots will find it easiest to do tasks in predictable, simpler, lower stakes environments. For example, robots are already doing a lot of warehouse jobs. This hasn’t yet decreased warehouse worker employment (perhaps because demand for warehouses has increased even faster with online shopping), but the next couple of generations of robotics could reach a tipping point.

5. Some closing thoughts on career strategy

Given these developments, how should you approach your next couple of career steps?

5.1 Look for ways to leapfrog entry-level white collar jobs

As AI increases the value of leadership skills, it’s decreasing the value of the entry-level jobs that previously served as a training path to them.

So as a college grad entering the job market who hoped to get one of these jobs, what should you do?

The ideal might be to find a role that lets you learn leadership skills right away (for instance, anywhere you can work with a good mentor), but what about if you can’t?

First, you can start to learn AI deployment and personal effectiveness skills in any job, and those are also high on my list.

Second, you might be able to find a way to start practicing leadership or communications skills in your existing role, perhaps just on a small scale (e.g. by managing a contractor, helping to launch a new product).

Otherwise you might be able to start some kind of side project or serious hobby, like running a voluntary community project, having a blog, or having a side business. These let you practice leadership skills, and by using AI tools you can achieve more faster than before.

In terms of full-time jobs, roles at small but growing organisations seem more attractive, because they let you work on these types of skills faster.

In contrast, in large companies, there’s more specialisation, which means the entry-level roles often involve more routine work.

If you have the option, roles at tech startups applying AI to a real problem seem especially attractive, since they let you learn about AI deployment, entrepreneurship, and generally getting shit done all at the same time. Here’s a write up of the case for moonshots.

If you’re not able to leapfrog the white collar path, then another option is to focus on sectors where performance is driven by complex physical skills, physical presence, and social skills (e.g. mediator, events organiser, luxury tourism).

5.2 Be cautious about starting long training periods, like PhDs and medicine

AI automation is already happening faster than previous technological waves,¹⁰ could speed up, and has hard-to-predict effects, making long training periods less attractive.

This isn’t to say you shouldn’t spend 1–2 years training, or even that you should never start long training programs. For example, graduate study could still be worth it due to a combination of (i) the value of true expertise going up, (ii) being able to do useful work during your studies, (iii) if you think AI progress will be slower, (iv) you lack other options. But it’s worth thinking harder about alternatives.

What about finishing college? For most people, this is still worth it because it still delivers a large boost in employability. However, the case for dropping out seems better than before (especially if your university doesn’t let you use AI tools). I usually caution against dropping out unless you already have an offer to do paid work. However, you could try to (i) get into a position where you might get such an offer faster (e.g. through summer projects) or (ii) finish college more quickly.

5.3 Make yourself more resilient to change

One way to deal with fast, unpredictable change is to learn the personal effectiveness skills that are useful in every job. But you can also think about ways to set your life up to be flexible and resilient:

• Not overly tying yourself to a single country, and living in a large city with many kinds of opportunities

• Saving more money than you would otherwise

• Investing in your general mental health

5.4 Ride the wave

The goal isn’t to find a single job that will always be resistant to automation, but rather to stay one or two steps ahead of it.

This means keeping on top of what AI is capable of, seeking out people to follow who have insights into what’s going on, and continually adjusting to where the biggest bottlenecks lie.

Take action

1. This week: find a small new way to apply AI in your current (or desired) job.

2. This month: choose one of the six skills, and think of 1–2 steps you could take to learn it faster.

3. This quarter: consider whether to make a larger change to focus more on these skills.

If you have questions about what this means for your career, comment below.

1) World-changing AI systems could come faster than expected

I’ve ranked AI as the most pressing global problem for over ten years, but it seems even more urgent today. In the last 1-2 years, I’ve pivoted to focus more on it, and I wish I’d pivoted more earlier.

There’s now a significant chance that AI which can contribute to scientific research or automate many jobs is created by 2030. Current systems can already do a lot, and there are clear ways to continue to improve them. Forecasters and experts widely agree the probability is much higher than it was even just a couple of years ago.

2) Society could be transformed – whether we’re ready or not

Lots of people hype AI as 'transformative' but few internalise how crazy it could really be. There's three different types of acceleration that could be possible, and are much more grounded in empirical research than a couple of years ago (and would render your current career plans obsolete):

1. The intelligence explosion: through feedback loops in algorithmic efficiency, it might only take a few years from developing advanced AI to having billions of AI remote workers, making cognitive labour available for pennies.

2. The technological explosion: estimates suggest that with sufficiently advanced AI 100 years of technological progress in 10 is plausible. That means we could have advanced biotech, robotics, novel political philosophies, and more arrive much sooner than commonly imagined.

3. The industrial explosion: if AI and robotics automate industrial production that would create a positive feedback loop, meaning production could plausibly end up doubling each year. Within a decade of reaching that growth rate, humanity would harvest all available solar energy on Earth and start to expand into space.

Along the way, we could also see rapid progress on many key technological challenges — like curing cancer and developing green energy. But…

3) Advanced AI could bring enormous dangers

It might be hard to keep control of billions of AI systems thinking 100x faster than ourselves. But that’s only the first hurdle. The developments above could also:

• Destabilise the world order (e.g. create conflict over Taiwan)

• Enable the development of new weapons of mass destruction, like man-made viruses

• Empower governments (or even individual companies) to entrench their power

• Force us to face civilisation-defining questions about how to treat AI systems, how to share the benefits of AI, and how to govern an expansion into space.

This isn’t just about ‘technical safety’, but about an entire range of downstream issues.

4) Under 10,000 people work full-time reducing the risks

Although it can feel like all anyone talks about is AI, only a few thousand people work full-time on navigating some of the most important aspects of the risks.

This is tiny compared to the millions working on more established issues like cancer or climate change, or the number of people trying to deploy the technology as quickly as possible.

If you switch to working on this issue now, you could be among the first 10,000 people helping humanity navigate what may be the one of the most important transitions in history.

5) There are more and more concrete jobs

A couple of years ago, there weren’t many clearly defined projects, positions or training routes to work on this issue. Today, there are more and more concrete ways to help, such as:

• This list of technical safety projects

• Joining one of the many growing AI policy think tanks around the world

• Improve forecasting and data about AI

• Building defences against man-made viruses, like better PPE and detection tools

• And more

80,000 Hours has compiled a list of 30+ important organisations, over 300 open jobs, and lists of fellowships, courses, internships, etc., to help you enter the field. Many of these are all well-paid too.

It’s true many of these jobs are extremely competitive, but due to their potential impact it could still be worth applying to them (while making sure you have a back-up plan).

You also don’t need to work in an explicitly “AI risk” focused organisation. For example there are hundreds of relevant government positions.

And otherwise you can contribute without changing job by donating, spreading clear thinking, building community around this issue, and investing in yourself to be ready to switch as more opportunities open up.

You don’t need to be technical or even focus directly on AI — we need people building organisations, in communications, and with many other skills. AI is going to affect every aspect of society, so people with knowledge of every aspect are needed (e.g. China, economics, biology, international governance, law, etc.).

The field was small until recently, so there’s comparatively few people with deep expertise. That means it’s often possible to spend about 100 hours reading and speaking to people, and transition in the field (and then keep learning from there). If you have a quantitative background, it’s possible to get to the technical forefront in under a year. The 80,000 Hours team can give you one-on-one advice on how to switch if you’re later-career, and how to skill-up if earlier. There’s more tactical advice here.

Real examples of people who switched:

• Rashida Polk was an experienced nurse, but wanted to switch to reducing pandemic risk. She applied to the Horizon Fellowship, and is working in a relevant Senate Committee.

• Neel Nanda studied maths and considered going into finance. He found out about AI risk and got an internship in the area. Now he leads research into interpretability at Google DeepMind.

• Katie Hearsum was working in banking, and transitioned an operations role at Longview Philanthropy, one of the largest funders in the space, and where she’s now the COO.

6) The next five years seem crucial

I’ve argued the chance of building powerful AI is unusually high between now and around 2030, and declines thereafter. This makes the next five years especially critical.

That creates an additional reason to switch soon:

• If transformative AI emerges in the next five years, you’ll be part of one of the most important transitions in human history.

• If it doesn’t (which is definitely a live possibility), you’ll have time to return to your previous path — while having learned about a technology that will still shape our world in significant ways.

The bottom line

If you’re fortunate enough to be able to find a role helping to navigate these risks (especially over the next 5–10 years), that’s probably the highest expected impact thing you can do.

But I don’t think everyone reading this should work on AI.

1. You might not have the flexibility to make a large career change right now. In that case, you could look to contribute from your current job and prepare to switch in the future — or like most people, you just might not have the luxury of making social impact your focus.

2. There are other important problems, and you might have far better fit for a job focused on one of them.

3. You might be too concerned about the (definitely huge) uncertainties about how best to help or be less convinced by the arguments that it’s pressing.

However, I’d encourage almost everyone who’s able to pursue an impactful career to seriously consider it. If you’re unsure you’ll be able to find something, keep in mind there’s a very wide range of approaches and opportunities, and they’re expanding all the time.

All this is why I’m writing a new guide to careers tackling AI. Read a summary with some more practical advice on how to switch:

Read now

If you’ve decided you’d like to focus on this issue, 80,000 Hours may be able to give you one-on-one advice and introductions to people in the field. APPLY NOW.1

Unfortunately, there aren’t.

There are only different groups of experts with different weaknesses.

This article is an overview of what five different types of experts say about when we’ll reach AGI, and what we can learn from them (that feeds into my full article on forecasting AI).

In short:

• Every group shortened their estimates in recent years.

• AGI before 2030 seems within the range of expert opinion, even if many disagree.

• None of the forecasts seem especially reliable, so they neither rule in nor rule out AGI arriving soon.

Here’s an overview of the five groups:

AI experts

1. Leaders of AI companies

The leaders of AI companies are saying that AGI arrives in 2–5 years, and appear to have recently shortened their estimates.

This is easy to dismiss. This group is obviously selected to be bullish on AI and wants to hype their own work and raise funding.

However, I don’t think their views should be totally discounted. They’re the people with the most visibility into the capabilities of next-generation systems, and the most knowledge of the technology.

And they’ve also been among the most right about recent progress, even if they’ve been too optimistic.

Most likely, progress will be slower than they expect, but maybe only by a few years.

2. AI researchers in general

One way to reduce selection effects is to look at a wider group of AI researchers than those working on AGI directly, including in academia. This is what Katja Grace did with a survey of thousands of recent AI publication authors.

The survey asked for forecasts of “high-level machine intelligence,” defined as when AI can accomplish every task better or more cheaply than humans. The median estimate was a 25% chance in the early 2030s and 50% by 2047 — with some giving answers in the next few years and others hundreds of years in the future.

The median estimate of the chance of an AI being able to do the job of an AI researcher by 2033 was 5%.¹

They were also asked about when they expected AI could perform a list of specific tasks (2023 survey results in red, 2022 results in blue).

Historically their estimates have been too pessimistic.

In 2022, they thought AI wouldn’t be able to write simple Python code until around 2027.

In 2023, they reduced that to 2025, but AI could maybe already meet that condition in 2023 (and definitely by 2024).

Most of their other estimates declined significantly between 2023 and 2022.

The median estimate for achieving ‘high-level machine intelligence’ shortened by 13 years.

This shows these experts were just as surprised as everyone else at the success of ChatGPT and LLMs. (Today, even many sceptics concede AGI could be here within 20 years, around when today’s college students will be turning 40.)

Finally, they were asked about when we should expect to be able to “automate all occupations,” and they responded with much longer estimates (e.g. 20% chance by 2079).

It’s not clear to me why ‘all occupations’ should be so much further in the future than ‘all tasks’ — occupations are just bundles of tasks. (In addition, the researchers think once we reach ‘all tasks,’ there’s about a 50% chance of an intelligence explosion.)

Perhaps respondents envision a world where AI is better than humans at every task, but humans continue to work in a limited range of jobs (like priests).² Perhaps they are just not thinking about the questions carefully.

Finally, forecasting AI progress requires a different skill set than conducting AI research. You can publish AI papers by being a specialist in a certain type of algorithm, but that doesn’t mean you’ll be good at thinking about broad trends across the whole field, or well calibrated in your judgements.

For all these reasons, I’m sceptical about their specific numbers.

My main takeaway is that, as of 2023, a significant fraction of researchers in the field believed that something like AGI is a realistic near-term possibility, even if many remain sceptical.

If 30% of experts say your airplane is going to explode, and 70% say it won’t, you shouldn’t conclude ‘there’s no expert consensus, so I won’t do anything.’

The reasonable course of action is to act as if there’s a significant explosion risk. Confidence that it won’t happen seems difficult to justify.

Expert forecasters

3. Metaculus

Instead of seeking AI expertise, we could consider forecasting expertise.

Metaculus aggregates hundreds of forecasts, which collectively have proven effective at predicting near-term political and economic events.

It has a forecast about AGI with over 1000 responses. AGI is defined with four conditions (detailed on the site).

As of December 2024, the forecasters average a 25% chance of AGI by 2027 and 50% by 2031.

The forecast has dropped dramatically over time, from a median of 50 years away as recently as 2020.

However, the definition used in this forecast is not great.

First, it’s overly stringent, because it includes general robotic capabilities. Robotics is currently lagging, so satisfying this definition could be harder than having an AI that can do remote work jobs or help with scientific research.

But the definition is also not stringent enough because it doesn’t include anything about long-horizon agency or the ability to have novel scientific insights.

An AI model could easily satisfy this definition but not be able to do most remote work jobs or help to automate scientific research.

Metaculus also seems to suffer from selection effects and their forecasts are seemingly drawn from people who are unusually into AI.

4. Superforecasters in 2022 (XPT survey)

Another survey asked 33 people who qualified as superforecasters of political events.

Their median estimate was a 25% chance of AGI (using the same definition as Metaculus) by 2048 — much further away.

However, these forecasts were made in 2022, before ChatGPT caused many people to shorten their estimates.

The superforecasters also lack expertise in AI, and they made predictions that have already been falsified about growth in training compute.

5. Samotsvety in 2023

In 2023, another group of especially successful superforecasters, Samotsvety, which has engaged much more deeply with AI, made much shorter estimates: ~28% chance of AGI by 2030 (from which we might infer a ~25% chance by 2029).

These estimates also placed AGI considerably earlier compared to forecasts they’d made in 2022.

More recently, one of the leaders of Samotsvety (Eli Lifland), was involved in a forecast for ‘superhuman coders’ as part of the AI 2027 project. This gave roughly a 25% chance of arriving in 2027.

However, compared to the superforecasters above, Samotsvety are selected for interest in AI.

Finally, all of the three groups of forecasters have been selected for being good at forecasting near-term current events, which could fail to generalise to forecasting long-term, radically novel events.

Summary of expert views on when AGI will arrive

In sum, it’s a confusing situation. Personally, I put some weight on all the groups, which averages me out at ‘experts think AGI before 2030 is a realistic possibility, but many think it’ll be much longer.’

This means AGI soon can’t be dismissed as ‘sci fi’ or unsupported by ‘real experts.’ Expert opinion can neither rule out nor rule in AGI soon.

Mostly, I prefer to think about the question bottom up, as I’ve done in my full article on when to expect AGI.

Learn more

• Why AGI might be here by 2030.

• Through a glass darkly by Scott Alexander is an exploration of what can be learned from expert forecasts on AI.

• ‘Long’ timelines to advanced AI have gotten crazy short by Helen Toner.

• Results of the largest survey of AI researchers from 2023, and some sceptical discussion of it.

• OpenAI’s Sam Altman: Shifted from saying in November “the rate of progress continues” to declaring in January “we are now confident we know how to build AGI”¹

• Anthropic’s Dario Amodei: Stated in January “I’m more confident than I’ve ever been that we’re close to powerful capabilities… in the next 2-3 years”

• Google DeepMind’s Demis Hassabis: Changed from “as soon as 10 years” in autumn to “probably three to five years away” by January.

Is it just hype? What explains the shift? And could we really have Artificial General Intelligence (AGI)² by 2028?

In this article, I interrogate these claims. I’ll examine what’s driven recent progress, estimate how far those drivers can continue, and explain why they’re likely to continue for at least four more years.

In particular, while in 2024 progress in LLM chatbots seemed to slow, a new approach started to work: teaching the models to reason using reinforcement learning.

In just a year, this let them surpass human PhDs at answering difficult scientific reasoning questions, and achieve expert-level performance on one-hour coding tasks.

We don’t know how capable AI will become, but extrapolating the recent rate of progress suggests that, by 2028, we could reach AI models with beyond-human reasoning abilities, expert-level knowledge in every domain, and that can autonomously complete multi-week projects, and progress would likely continue from there.

No longer mere chatbots, these ‘agent’ models might soon satisfy many people’s definitions of AGI — roughly, AI systems that match human performance at most knowledge work (see full def in footnotes).²

This means that, while the company leaders are probably overoptimistic, there’s enough evidence to take their position very seriously.

Where we draw the ‘AGI’ line is ultimately arbitrary. What matters is these models could start to accelerate AI research itself, unlocking vastly greater numbers of more capable ‘AI workers’. In turn, sufficient automation could trigger explosive growth and 100 years of scientific progress in 10 — a transition society isn’t prepared for.

While this might sound outlandish, it’s within the range of possibilities many experts think is possible. This article aims to give you a primer on what you need to know to understand why, and also the best arguments against.

I’ve been writing about AGI since 2014. Back then, AGI arriving within five years seemed very unlikely. Today, the situation seems dramatically different. We can see the outlines of how it could work and who will build it.

In fact, the next five years seem unusually crucial. The basic drivers of AI progress — investments in computational power and algorithmic research — cannot continue increasing at current rates much beyond 2030. That means we either reach AI systems capable of triggering an acceleration soon, or progress will most likely slow significantly.

Either way, the next five years are when we’ll find out. Let’s see why.

This is part of my new AGI careers guide. Sign up to receive future articles.

In a nutshell

• Four key factors are driving AI progress: larger base models, teaching models to reason, increasing models’ thinking time, and building agent scaffolding for multi-step tasks. These are underpinned by increasing computational power to run and train AI systems, as well as increasing human capital going into algorithmic research.

• All of these drivers are set to continue until 2028 and perhaps until 2032.

• This means we should expect major further gains in AI performance. We don’t know how large they’ll be, but extrapolating recent trends on benchmarks suggests we’ll reach systems with beyond-human performance in coding and scientific reasoning, and that can autonomously complete multi-week projects.

• Whether we call these systems’AGI’ or not, they could be sufficient to enable AI research itself, robotics, the technology industry, and scientific research to accelerate, leading to transformative impacts.

• Alternatively, AI might fail to overcome issues with ill-defined, high-context work over long time horizons and remain a tool (even if much improved compared to today).

• Increasing AI performance requires exponential growth in investment and the research workforce. At current rates, we will likely start to reach bottlenecks around 2030. Simplifying a bit, that means we’ll likely either reach AGI by around 2030 or see progress slow significantly. Hybrid scenarios are also possible, but the next five years seem especially crucial.

I. What’s driven recent AI progress? And will it continue?

The deep learning era

In 2022, Yann LeCun, the chief AI scientist at Meta and a Turing Award winner, said:

“I take an object, I put it on the table, and I push the table. It’s completely obvious to you that the object will be pushed with the table…There’s no text in the world I believe that explains this. If you train a machine as powerful as could be…your GPT-5000, it’s never gonna learn about this.”

And, of course, if you plug this question into GPT-4 it has no idea how to answer:

Just kidding. Within a year of LeCun’s statement, here’s GPT-4.

And this isn’t the only example of experts being wrongfooted.

Before 2011, AI was famously dead.

But that totally changed when conceptual insights from the 1970s and 1980s combined with massive amounts of data and computing power to produce the deep learning paradigm.

Since then, we’ve repeatedly seen AI systems going from total incompetence to greater-than-human performance in many tasks within a couple of years.

For example, in 2022, if you asked Midjourney to draw “an otter on a plane using wifi,” this was the result:

Two years later, you could get this with Veo 2:

In 2019, GPT-2 could just about stay on topic for a couple of paragraphs. And that was considered remarkable progress.

Critics like LeCun were quick to point out that GPT-2 couldn’t reason, show common sense, exhibit understanding of the physical world, and so on. But many of these limitations were overcome within a couple of years.

Over and over again, it’s been dangerous to bet against deep learning. Today, even LeCun says he expects AGI in “several years.”²

The limitations of current systems aren’t what to focus on anyway. The more interesting question is: where this might be heading? What explains the leap from GPT-2 to GPT-4, and will we see another?

What’s coming up

At the broadest level, AI progress has been driven by:

• More computational power

• Better algorithms

Both are improving rapidly.

More specifically, we can break recent progress down into four key drivers:

1. Scaling pretraining to create a base model with basic intelligence

2. Using reinforcement learning to teach the base model to reason

3. Increasing test-time compute to increase how long the model thinks about each question

4. Building agent scaffolding so the model can complete complex tasks

In the rest of this section, I’ll explain how each of these works and try to project them forward. Delve (ahem) in, and you’ll understand the basics of how AI is being improved.

In section two I’ll use this to forecast future AI progress, and finally explain why the next five years are especially crucial.

1. Scaling pretraining to create base models with basic intelligence

Pretraining compute

People often imagine that AI progress requires huge intellectual breakthroughs, but a lot of it is more like engineering. Just do (a lot) more of the same, and the models get better.

In the leap from GPT-2 to GPT-4, the biggest driver of progress was just applying dramatically more computational power to the same techniques, especially to ‘pretraining.’

Modern AI works by using artificial neural nets, involving billions of interconnected parameters organised into layers. During pretraining (a misleading name, which simply indicates it’s the first type of training), here’s what happens:

1. Data is fed into the network (such as an image of a cat).

2. The values of the parameters convert that data into a predicted output (like a description: ‘this is a cat’).

3. The accuracy of those outputs is graded vs. reference data.

4. The model parameters are adjusted in a way that’s expected to increase accuracy.

5. This is repeated over and over, with trillions of pieces of data.

This method has been used to train all kinds of AI, but it’s been most useful when used to predict language. The data is text on the internet, and LLMs are trained to predict gaps in the text.

More computational power for training (i.e. ‘training compute’) means you can use more parameters, which lets the models learn more sophisticated and abstract patterns in the data. It also means you can use more data.

Since we entered the deep learning era, the number of calculations used to train AI models has been growing at a staggering rate — more than 4x per year.

This was driven by spending more money and using more efficient chips.³

Historically, each time training compute has increased 10x, there’s been a steady gain in performance across many tasks and benchmarks.

For example, as training compute has grown a thousandfold, AI models have steadily improved at answering diverse questions—from commonsense reasoning to understanding social situations and physics. This is demonstrated on the ‘BIG-Bench Hard’ benchmark, which features diverse questions specifically chosen to challenge LLMs:

Likewise, OpenAI created a coding model that could solve simple problems, then used 100,000 times more compute to train an improved version. As compute increased, the model correctly answered progressively more difficult questions.⁴

These test problems weren’t in the original training data, so this wasn’t merely better search through memorised problems.

This relationship between training compute and performance is called a ‘scaling law.’⁵

Papers about these laws had been published by 2020. To those following this research, GPT-4 wasn’t a surprise — it was just a continuation of a trend.

Algorithmic efficiency

Training compute has not only increased, but researchers have found far more efficient ways to use it.

Every two years, the compute needed to get the same performance across a wide range of models has decreased tenfold.

These gains also usually make the models cheaper to run. DeepSeek-V3 was promoted as a revolutionary efficiency breakthrough, but it was roughly on trend: released two years after GPT-4, it’s about 10 times more efficient.⁶

Algorithmic efficiency means that, not only is four times as much compute used on training each year, but that compute also goes three times further. The two multiply together to produce a 12 times increase in ‘effective’ compute each year.

That means the chips that were used to train GPT-4 in three months could have been used to train a model with the performance of GPT-2 about 300,000 times over.⁷

This increase in effective compute took us from a model that could just about string some paragraphs together to GPT-4 being able to do things like:

• Beat most high schoolers at college entrance exams

• Converse in natural language — in the long-forgotten past this was considered a mark of true intelligence, a la the Turing test

• Solve the Winograd schemas — a test of commonsense reasoning that in the 2010s was regarded as requiring true understanding⁸

• Create art that most people can’t distinguish from the human-produced stuff⁹

How much further can pretraining scale?

If current trends continue, then by around 2028, someone will have trained a model with 300,000 times more effective compute than GPT-4.¹⁰

That’s the same increase we saw from from GPT-2 to GPT-4, so if spent on pretraining, we could call that hypothetical model ‘GPT-6.’¹¹

After a pause in 2024, GPT-4.5-sized models appear to be on trend, and companies are already close to GPT-5-sized models, which forecasters expect to be released in 2025.

But can this trend continue all the way to GPT-6?

The CEO of Anthropic, Dario Amodei, projects GPT-6-sized models will cost about $10bn to train.¹² That’s still affordable for companies like Google, Microsoft, or Meta, which earn $50–100bn in profits annually.¹³

In fact, these companies are already building data centres big enough for such training runs¹⁴ — and that was before the $100bn+ Stargate project was announced.

Frontier AI models are also already generating over $10bn of revenue,¹⁵ and revenue has been more than tripling each year, so AI revenue alone will soon be enough to pay for a $10bn training run.

I’ll discuss the bottlenecks more later but the most plausible one is training data. However, the best analysis I’ve found suggests that there will be enough data to carry out a GPT-6 scale training run by 2028.

And even if this isn’t the case, it’s no longer crucial — the AI companies have discovered ways to circumvent the data bottleneck.

2. Post training of reasoning models with reinforcement learning

People often say “ChatGPT is just predicting the next word.” But that’s never been quite true.

Raw prediction of words from the internet produces outputs that are regularly crazy (as you might expect, given that it’s the internet).

GPT only became truly useful with the addition of reinforcement learning from human feedback (RLHF):

1. Outputs from the ‘base model’ are shown to human raters.

2. The raters are asked to judge which are most useful.

3. The model is adjusted to produce more outputs like the helpful ones (‘reinforcement’).

A model that has undergone RLHF isn’t just ‘predicting the next token,’ it’s been trained to predict what human raters find most helpful.

You can think of the initial LLM as providing a foundation of conceptual structure. RLHF is essential for directing that structure towards a particular useful end.

RHLF is one form of ‘post training,’ named because it happens after pretraining (though both are simply types of training).

There are many other kinds of post training enhancements, including things as simple as letting the model access a calculator or the internet. But there’s one that’s especially crucial right now: reinforcement learning to train the models to reason.

This idea is that instead of training the model to do what humans find helpful, it’s trained to correctly answer problems. Here’s the process:

1. Show the model a problem with a verifiable answer, like a math puzzle.

2. Ask it to produce a chain of reasoning to solve the problem (‘chain of thought’).¹⁶

3. If the answer is correct, adjust the model to be more like that (‘reinforcement’).¹⁷

4. Repeat.

This process teaches the LLM to construct long chains of (correct) reasoning about logical problems.

Before 2023, this didn’t seem to work. If each step of reasoning is too unreliable, then the chains quickly go wrong. And if you can’t get close to the answer, then you can’t give it any reinforcement.

But in 2024, as many were saying AI progress had stalled, this new paradigm started to take off.

Consider the GPQA Diamond benchmark — a set of scientific questions designed so that people with PhDs in the field can mostly answer them, but non-experts can’t, even with 30 minutes of access to Google. It contains questions like this:

In 2023, GPT-4 performed only slightly better than random guessing on this benchmark. It could handle the reasoning required for high school-level science problems, but couldn’t manage PhD-level reasoning.

However, in October 2024, OpenAI took the GPT-4o base model and used reinforcement learning to create o1.¹⁸

It achieved 70% accuracy — making it about equal to PhDs in each field at answering these questions.

It’s no longer tenable to claim these models are just regurgitating their training data — neither the answers nor the chains of reasoning required to produce them exist on the internet.

Most people aren’t answering PhD-level science questions in their daily life, so they simply haven’t noticed recent progress. They still think of LLMs as basic chatbots.

But o1 was just the start. At the beginning of a new paradigm, it’s possible to get gains especially quickly.

Just three months after o1, OpenAI released results from o3. It’s the second version, named ‘o3’ because ‘o2’ is a telecom company. (But please don’t ask me to explain any other part of OpenAI’s model-naming practices.)

o3 is probably o1 but with even more reinforcement learning (and another change I’ll explain shortly).

It surpassed human expert-level performance on GPQA:

Reinforcement should be most useful for problems that have verifiable answers, such as in science, math, and coding.¹⁹ o3 performs much better in all of these areas than its base model.

Most benchmarks of math questions have now been saturated — leading models can get basically every question right.

In response, Epoch AI created Frontier Math — a benchmark of insanely hard mathematical problems. The easiest 25% are similar to Olympiad-level problems. The most difficult 25% are, according to Fields Medalist Terence Tao, “extremely challenging,” and would typically need an expert in that branch of mathematics to solve them.

Previous models, including GPT-o1, could hardly solve any of these questions.²⁰ In December 2024, OpenAI claimed that GPT-o3 could solve 25%.²¹

These results went entirely unreported in the media. On the very day of the o3 results announcement, The Wall Street Journal was running this story:

This misses the crucial point that GPT-5 is no longer necessary — a new paradigm has started, which can make even faster gains than before.

How far can scaling reasoning models continue?

In January, DeepSeek replicated many of o1’s results. Their paper revealed that even basically the simplest version of the process works, suggesting there’s a huge amount more to try.

DeepSeek-R1 also reveals its entire chain of reasoning to the user, demonstrating its sophistication and surprisingly human quality: it’ll reflect on its answers, backtrack when wrong, consider multiple hypotheses, have insights, and more.

All of this behaviour emerges out of simple reinforcement learning. OpenAI researcher Sabastian Bubeck observed:

“No tactic was given to the model. Everything is emergent. Everything is learned through reinforcement learning. This is insane.”

The compute for the reinforcement learning stage of training DeepSeek-R1 likely only cost about $1m.

If it keeps working, OpenAI, Anthropic, and Google could now spend $1bn on the same process, approximately a 1000x scale up of compute.²²

One reason it’s possible to scale up this much is that the models generate their own data.

This might sound circular, and the idea that synthetic data causes ‘model collapse‘ has been widely discussed.

But there’s nothing circular in this case. You can ask GPT-o1 to solve 100,000 math problems, then take only the cases where it got the right answer, and use them to train the next model.

Because the solutions can be quickly verified, you’ve generated more examples of genuinely good reasoning.

In fact, this data is much higher quality than what you’ll find on the internet because it contains the whole chain of reasoning and is known to be correct (not something the internet is famous for).²³

This potentially creates a flywheel:

1. Have your model solve a bunch of problems.

2. Use the solutions to train the next model.²⁴

3. The next model can solve even harder problems.

4. That generates even more solutions.

5. And so on.

If the models can already perform PhD-level reasoning, the next stage would be researcher-level reasoning, and then generating novel insights.

This likely explains the unusually optimistic statements from AI company leaders. Sam Altman’s shift in opinion coincides exactly with the o3 release in December 2024.

Although most powerful in verifiable domains, the reasoning skills developed will probably generalise at least a bit. We’ve already seen o1 improve at legal reasoning, for instance.²⁵

In other domains like business strategy or writing, it’s harder to clearly judge success, so the process takes longer, but we should expect it to work to some degree. How well this works is a crucial question going forward.

3. Increasing how long models think

If you could only think about a problem for a minute, you probably wouldn’t get far.

If you could think for a month, you’ll make a lot more progress — even though your raw intelligence isn’t higher.

LLMs used to be unable to think about a problem for more than about a minute before mistakes compounded or they drifted off topic, which really limited what they could do.

But as models have become more reliable at reasoning, they’ve become better at thinking for longer.

OpenAI showed that you can have o1 think 100 times longer than normal and get linear increases in accuracy on coding problems.

This is called using ‘test time compute’ – compute spent when the model is being run rather than trained.

If GPT-4o could usefully think for about one minute, GPT-o1 and DeepSeek-R1 seem like they can think for the equivalent of about an hour.²⁶

As reasoning models get more reliable, they will be able to think for longer and longer.

At current rates, we’ll soon have models that can think for a month — and then a year.

(It’s particularly intriguing to consider what happens if they can think indefinitely—given sufficient compute, and assuming progress is possible in principle, they could continuously improve their answers to any question.)

Using more test time compute can be used to solve problems via brute force. One technique is to try to solve a problem 10, 100, or 1000 times, and to pick the solution with the most ‘votes’. This is probably another way o3 was able to beat o1.²⁷

The immediate practical upshot of all this is you can pay more to get more advanced capabilities earlier.

Quantitatively, in 2026, I expect you’ll be able to pay 100,000 times more to get performance that would have previously only been accessible in 2028.²⁸

Most users won’t be willing to do this, but if you have a crucial engineering, scientific, or business problem, even $1m is a bargain.

In particular, AI researchers may be able to use this technique to create another flywheel for AI research. It’s a process called iterated distillation and amplification, which you can read about here. Here’s roughly how it would work:

1. Have your model think for longer to get better answers (‘amplification’).

2. Use those answers to train a new model. That model can now produce almost the same answers immediately without needing to think for longer (‘distillation’).

3. Now have the new model think for longer. It’ll be able to generate even better answers than the original.

4. Repeat.

This process is essentially how DeepMind made AlphaZero superhuman at Go within a couple of days, without any human data.

4. The next stage: building better agents

GPT-4 resembles a coworker on their first day who is smart and knowledgeable, but who only answers a question or two before leaving the company.

Unsurprisingly, that’s also only a bit useful.

But the AI companies are now turning chatbots into agents.

An AI ‘agent’ is capable of doing a long chain of tasks in pursuit of a goal.

For example, if you want to build an app, rather than asking the model for help with each step, you simply say, “Build an app that does X.” It then asks clarifying questions, builds a prototype, tests and fixes bugs, and delivers a finished product — much like a human software engineer.

Agents work by taking a reasoning model and giving it a memory and access to tools (a ‘scaffolding’):

1. You tell the reasoning module a goal, and it makes a plan to achieve it.

2. Based on that, it uses the tools to take some actions.

3. The results are fed back into the memory module.

4. The reasoning module updates the plan.

5. The loop continues until the goal is achieved (or determined not possible).

AI agents already work a bit.

SWE-bench Verified is a benchmark of real-world software engineering problems from GitHub that typically take about an hour to complete.

GPT-4 basically can’t do these problems because they involve using multiple applications.

However, when put into a simple agent scaffolding:²⁹

• GPT-4 can solve about 20%.

• Claude Sonnet 3.5 could solve 50%.

• And GPT-o3 reportedly could solve over 70%.

This means o3 is basically as good as professional software engineers at completing these discrete tasks.

On competition coding problems, it would have ranked about top 200 in the world.

Here’s how these coding agents look in action:

Now consider perhaps the world’s most important benchmark: METR’s set of difficult AI research engineering problems (‘RE Bench’).

These include problems, like fine-tuning models or predicting experimental results, that engineers tackle to improve cutting-edge AI systems. They were designed to be genuinely difficult problems that closely approximate actual AI research.

A simple agent built on GPT-o1 and Claude 3.5 Sonnet is better than human experts when given two hours.

This performance exceeded the expectations of many forecasters (and o3 hasn’t been tested yet).³⁰

AI performance increases more slowly than human performance when given more time, so human experts still surpass the AIs at around the four hour mark.

But the AI models are catching up fast.

GPT-4o was only able to do tasks which took humans about 30 minutes.³²

METR made a broader benchmark of computer use tasks categorised by time horizon. GPT-2 was only able to do tasks that took humans a few seconds; GPT-4 managed a few minutes; and the latest reasoning models could do tasks that took humans just under an hour.

If this trend continues to the end of 2028, AI will be able to do AI research & software engineering tasks that take several weeks as well as many human experts.

The orange line shows that the trend in the last year has been even faster, perhaps due to the reasoning models paradigm.

Update April 2025: After this article was first published, results for o3 were released and it appears to be on the faster post-2024 trend rather than the slower post-2020 trend discussed above. If this continues, then progress would be almost twice as fast: time horizon doubling every four months rather than every seven. If this faster trend is indeed due to the scale up of reinforcement learning, it probably can’t continue at recent rates for more than 1-2 years, so we might see another 1-2 years of 4 month doubling times, followed by a reversion to the previous 7 month trend. Alternatively, this could be the start of a positive feedback loop, leading to hyperexponential progress.

AI models are also increasingly understanding their context — correctly answering questions about their own architecture, past outputs, and whether they’re being trained or deployed — another precondition for agency.

On a lighter note, while Claude 3.7 is still terrible at playing Pokemon, it’s much better than 3.5, and just a year ago, Claude 3 couldn’t play at all.

These graphs above explain why, although AI models can be very ‘intelligent’ at answering questions, they haven’t yet automated many jobs.

Most jobs aren’t just lists of discrete one hour tasks –– they involve figuring out what to; do coordinating with a team; long, novel projects with a lot of context, etc.

Even in one of AI’s strongest areas — software engineering –– today it can only do tasks that take under an hour. And it’s still often tripped up by things like finding the right button on a website. This means it’s a long way from being able to fully replace software engineers.

However, the trends suggest there’s a good chance that soon changes. An AI that can do 1-day or 1-week tasks would be able to automate dramatically more work than current models. Companies could start to hire hundreds of ‘digital workers’ overseen by a small number of humans.

How far can the trend of improving agents continue?

OpenAI dubbed 2025 the “year of agents.”

• While AI agent scaffolding is still primitive, it’s a top priority for the leading labs, which should lead to more progress.

• Gains will also come from hooking up the agent scaffolding to ever more powerful reasoning models — giving the agent a better, more reliable ‘planning brain.’

• Those in turn will be based on base models that have been trained on a lot more video data, which might make the agents much better at perception — a major bottleneck currently.

Once agents start working a bit, that unlocks more progress:

• Set an agent a task, like making a purchase or writing a popular tweet. Then if it succeeds, use reinforcement learning to make it more likely to succeed next time.

• In addition, each successfully completed task can be used as training data for the next generation of agents.

The world is an unending source of data, which lets the agents naturally develop a causal model of the world.³²

Any of these measures could significantly increase reliability, and as we’ve seen several times in this article, reliability improvements can suddenly unlock new capabilities:

• Even a simple task like finding and booking a hotel that meets your preferences requires tens of steps. With a 90% chance of completing each step correctly, there’s only a 10% chance of completing 20 steps correctly.

• However with 99% reliability per step, the overall chance of success leaps from 10% to 80% — the difference between not useful to very useful.

So progress could feel quite explosive.

All this said, agency is the most uncertain of the four drivers. We don’t yet have great benchmarks to measure it, so while there might be a lot of progress at navigating certain types of task, progress could remain slow on other dimensions. A few significant areas of weakness could hamstring AI’s applications. More fundamental breakthroughs might be required to make it really work.

None-the-less, recent trends and the above improvements in the pipeline mean I expect to see significant progress.

II. How good will AI become by 2030?

The four drivers projected forwards

Let’s recap everything we’ve covered so far. Looking ahead at the next two years, all four drivers of AI progress seem set to continue and build on each other:

1. A base model trained with 500x more effective compute than GPT-4 will be released (‘GPT-5’).

2. That model could be trained to reason with up to 100x more compute than o1 (‘o5’).

3. It’ll be able to think for the equivalent of a month per task when needed.

4. It’ll be hooked up to an improved agent scaffolding and further reinforced to be more agentic.

And that won’t be the end. The leading companies are on track to carry out $10bn training runs by 2028. This would be enough to pretrain a GPT-6-sized base model and do 100x more reinforcement learning (or some other combination).³³

In addition, new drivers like reasoning models appear roughly every 1–2 years, so we should project at least one more discovery like this in the next four years. And there’s some chance we might see a more fundamental advance more akin to deep learning itself.

Putting all this together, people who picture the future as ‘slightly better chatbots’ are making a mistake. Absent a major disruption,³⁶ progress is not going to plateau here.

The multi-trillion dollar question is how advanced AI will get.

Trend extrapolation of AI capabilities

Ultimately no-one knows, but one way to get a more precise answer is to extrapolate progress on benchmarks measuring AI capabilities.

Since all the drivers of progress are continuing at similar rates to the past, we can roughly extrapolate the recent rate of progress.³⁷

Here’s a summary of all the benchmarks we’ve discussed (plus a couple of others) and where we might expect them to be in 2026:

This implies that in two years we should expect AI systems that:

• Have expert-level knowledge of every field

• Can answer math and science questions as well as many professional researchers

• Are better than humans at coding

• Have general reasoning skills better than almost all humans

• Can autonomously complete many day long tasks on a computer

• And are still rapidly improving

The next leap might take us into beyond-human-level problem solving — the ability to answer as-yet-unsolved scientific questions independently.

What jobs would these systems be able to help with?

Many bottlenecks hinder real-world AI agent deployment, even for those that can use computers. These include regulation, reluctance to let AIs make decisions, insufficient reliability, institutional inertia, and lack of physical presence.⁴¹

Initially, powerful systems will also be expensive, and their deployment will be limited by available compute, so they will be directed only at the most valuable tasks.

This means most of the economy will probably continue pretty much as normal for a while. You’ll still consult human doctors (even if they use AI tools), get coffee from human baristas, and hire human plumbers.

However, there are a few crucial areas where, despite these bottlenecks, these systems could be rapidly deployed with significant consequences.

Software engineering

This is where AI is being most aggressively applied today. Google has said about 25% of their new code is written by AIs. Y Combinator startups say it’s 95%, and that they’re growing several times faster than before.

If coding becomes 10x cheaper, we’ll use far more of it. Maybe fairly soon, we’ll see billion-dollar software startups with a small number of human employees and hundreds of AI agents. Several AI startups have already become the fastest-growing companies of all time.

So this narrow application of AI could produce hundreds of billions of dollars of economic value pretty quickly — sufficient to fund continued AI scaling.

AI’s application to the economy could expand significantly from there. For instance, Epoch estimate that perhaps a third of work tasks can be performed remotely through a computer, and automation of those could more than double the economy.

Scientific research

The creators of AlphaFold already won the Nobel Prize for designing an AI that solves protein folding.

AI models have also found hundreds of thousands stable crystals that could be used in material science and created faster and more accurate weather forecasts.⁴² I expect many more results like this once scientists have adapted AI to solve specific problems, for instance by training on genetic or cosmological data.

Future models might be able to have genuinely novel insights simply by someone asking them. But, even if not, a lot of science is amenable to brute force. In particular, in any domain that’s mainly virtual but has verifiable answers — such as mathematics, economic modeling, theoretical physics, or computer science — research could be accelerated by generating thousands of ideas and then verifying which ones work.

Even an experimental field like biology is also bottlenecked by things like programming and data analysis, constraints that could be substantially alleviated.

A single invention like nuclear weapons can change the course of history, so the impact of any speed up here could be dramatic.

AI research

A field that’s especially amenable to acceleration is AI research itself. Besides being fully virtual, it’s the field that AI researchers understand best, have huge incentives to automate, and face no barriers to deploying AI.

Initially, this will look like researchers using ‘intern-level’ AI agents to unblock them on specific tasks or software engineering capacity (which is a major bottleneck), or even help brainstorm ideas.

Later, it could look like having the models read all the literature, generate thousands of ideas to improve the algorithms, and automatically test them in small-scale experiments.

An AI model has already produced an AI research paper that was accepted to a conference workshop. Here’s a list of other ways AI is already being applied to AI research.

Given all this, it’s plausible we’ll have AI agents doing AI research before people have figured out all the kinks that enable AI to do most remote work jobs.

Broad economic application of AI is therefore not necessarily a good way to gauge AI progress — it may follow explosively after AI capabilities have already advanced substantially.

What’s the case against impressive AI progress by 2030?

Here’s the strongest case against in my mind.

First, concede that AI will likely become superhuman at clearly defined, discrete tasks, which means we’ll see continued rapid progress on benchmarks.

But argue it’ll remain poor at ill-defined, high-context, and long-time-horizon tasks.

That’s because these kinds of tasks don’t have clearly and quickly verifiable answers, and so they can’t be trained with reinforcement learning, and they’re not in the training data either.

That means the rate of progress on these kinds of tasks will be slow, and might even hit a plateau.

If you also argue its starting position is weak, then even after 4-6 more years of progress it still might be bad. The METR data shows AI can’t complete many computer use tasks that humans find trivial to do in a couple of minutes, especially at high reliability, and it’s still worse than a 7 year old child at Pokemon.

Second, argue that most knowledge jobs consist significantly of these long-horizon, messy, high-context tasks.

For example, software engineers spend a lot of their time figuring out what to build, coordinating with others, and understanding massive code bases rather than knocking off a list of well-defined tasks. Even if their productivity at coding increases 10x, if coding is only 50% of their work, their overall productivity only roughly doubles.

A prime example of a messy, ill-defined task is having novel conceptual insights, so you could argue this task, which is especially important for unlocking an acceleration, is likely to be the hardest to automate (contrary to others who think AI research might be easier to automate than many other jobs).

In this scenario, we’ll have extremely smart and knowledgeable AI assistants, and perhaps an acceleration in some limited virtual domains (perhaps like mathematics research), but they’ll remain tools, and humans will remain the main economic & scientific bottleneck.

Human AI researchers will see their productivity increase but not enough to start a positive feedback loop – AI progress will remain bottlenecked by novel insights, human coordination, and compute.

These limits, combined with problems finding a business model and the other barriers to deploying AI, will mean the models won’t create enough revenue to justify training runs over $10bn. That’ll mean progress slows massively after about 2028.⁴² Once progress slows, the profit margins on frontier models collapse, making it even harder to pay for more training.

The primary counterargument is the earlier graph from METR: models are improving at acting over longer horizons, which requires deeper contextual understanding and handling of more abstract, complex tasks. Projecting this trend forward suggests much more autonomous models within four years.

This could be achieved via many incremental advances I’ve sketched,⁴³ but it’s also possible we’ll see a more fundamental innovation arise — the human brain itself proves such capabilities are possible.

Moreover, long horizon tasks can most likely be broken down into shorter tasks (e.g. making a plan, executing the first step etc.). If AI gets good enough at shorter tasks, then long horizon tasks might rapidly start to work too.

This is perhaps the central question of AI forecasting right now: will the horizon over which AIs can act plateau or continue to improve?

Here are some other ways AI progress could be slower or unimpressive:

• Disembodied cognitive labour could turn out not to be very useful, even in science, since innovation arises out of learning by doing across the economy. Broader automation (which will take much longer) is required. Read more.

• Pretraining could have big diminishing returns, so GPT-5 and GPT-6 will be disappointing (perhaps due to diminishing data quality).

• AI will continue to be bad at visual perception, limiting its ability to use a computer (see Moravec’s paradox). More generally, AI capabilities could remain very spiky – weak on dimensions that aren’t yet well understood, and this could limit their application.

• Benchmarks could seriously overstate progress due to issues with data contamination, and the difficulty of capturing messy tasks.

• An economic crisis, Taiwan conflict, other disaster, or massive regulatory crackdown could delay investment by several years.

• There are other unforeseen bottlenecks (cf planning fallacy).

For deeper exploration of the skeptical view, see “Are we on the brink of AGI?” by Steve Newman, “The promise of reasoning models” by Matthew Barnnett, “A bear case: My predictions regarding AI progress,” by Thane Ruthenis, and this podcast debate with Epoch AI.

Ultimately, the evidence will never be decisive one way or another, and estimates will rely on judgement calls over which people can reasonably differ. However, I find it hard to look at the evidence and not put significant probability on AGI by 2030.

When do the ‘experts’ expect AGI to arrive?

I’ve made some big claims. As a non-expert, it would be great if there were experts who could tell us what to think.

Unfortunately, there aren’t. There are only different groups, with different drawbacks.

I’ve reviewed the views of these different groups of experts in a separate article.

One striking point is that every group has shortened their estimates dramatically. Today even many AI ‘skeptics’ think AGI will be achieved in 20 years – mid career for today’s college students.

My overall read is that AGI by 2030 is within scope of expert opinion, so dismissing it as ‘sci fi’ is unjustified. Indeed, the people who know the most about the technology seem to have the shortest timelines.

Of course many experts think it’ll take much longer. But if 30% of experts think a plane will explode, and the other 70% think it’ll be fine, as non-experts we shouldn’t conclude it definitely won’t. If something is uncertain, that doesn’t mean it won’t happen.

III. Why the next 5 years are crucial

It’s natural to assume that since we don’t know when AGI will emerge, it might arrive soon, in the 2030s, the 2040s, and so on.

Although it’s a common perspective, I’m not sure it’s right.

The core drivers of AI progress are more compute and better algorithms.

More powerful AI is most likely to be discovered when the compute and labour used to improve AIs is growing most dramatically.

Right now, the total compute available for training and running AI is growing 3x per year,⁴⁴ and the workforce is growing rapidly too.

This means that each year, the number of AI models that can be run increases 3x. In addition, three times more compute can be used for training, and that training can use better algorithms, which means they get more capable as well as more numerous.

Earlier, I argued these trends can continue until 2028. But now I’ll show it most likely runs into bottlenecks shortly thereafter.

Bottlenecks around 2030

First, money:

• Google, Microsoft, Meta etc. are spending tens of billions of dollars to build clusters that could train a GPT-6-sized model in 2028.

• Another 10x scale up would require hundreds of billions of investment. That’s do-able, but more than their current annual profits and would be similar to another Apollo Program or Manhattan Project in scale.⁴⁵

• GPT-8 would require trillions. AI would need to become a top military priority or already be generating trillions of dollars of revenue (which would probably already be AGI).

Even if the money is available there will also be bottlenecks such as:

• Power: Current levels of AI chip sales, if sustained, mean that AI chips will use 4%+ of US electricity by 2028⁴⁶, but another 10x scale up would be 40%+. This is possible, but it would require building a lot of power plants.

• Chip production: Taiwan Semiconductor Manufacturing Company (TSMC) manufactures all of the world’s leading AI chips, but its most advanced capacity is still mostly used for mobile phones. That means TSMC can comfortably produce 5x more AI chips than it does now. However, reaching 50x would be a huge challenge. ⁴⁷

• ‘Latency limitations‘ could also prevent training runs as large as GPT-7.⁴⁸

So most likely, the rate of growth in compute slows around 2028–2032.

Algorithmic progress is also very rapid right now, but as each discovery gets made, the next one becomes harder and harder. Maintaining a constant rate of progress requires an exponentially growing research workforce.

In 2021, OpenAI had about 300 employees; today, it has about 3,000. Anthropic and DeepMind have also grown more than 3x, and new companies have entered. The number of ML papers produced per year has roughly doubled every two years.⁴⁹

It’s hard to know exactly how to define the workforce of people who are truly advancing capabilities (vs selling the product or doing other ML research). But if the workforce needs to double every 1–3 years, that can only last so long before the talent pool runs out.⁵⁰

My read is that growth can easily continue to the end of the decade but will probably start to slow in the early 2030s (unless AI has become good enough to substitute for AI researchers by then).

Algorithmic progress also depends on increasing compute, which enables more experiments. With sufficient compute, researchers can even conduct brute force searches for optimal algorithms. Thus, slowing compute growth will correspondingly slow algorithmic progress.

If compute and algorithmic efficiency increase by just 50% annually rather than 3x, a leap equivalent to the leap from GPT-3 to GPT-4 would take over 14 years instead of 2.5.

It also reduces the probability of discovering a new AI paradigm.

So there’s a race:

• Can AI models improve enough to generate enough revenue to pay for their next round of training before it’s no longer affordable?

• Can the models start to contribute to algorithmic research before we run out of human researchers thrown at the problem?

The moment of truth will be around 2028–2032.

Either progress slows, or AI itself overcomes these bottlenecks, allowing progress to continue or even accelerate.

Two potential futures for AI

If AI capable of contributing to AI research isn’t achieved before 2028–2032, the annual probability of its discovery decreases substantially.

Progress won’t suddenly halt — it’ll slow more gradually. Here are some illustrative estimates of probability of reaching AGI (don’t quote me on the exact numbers!):

Very roughly, we can plan for two scenarios:⁵¹

1. Either we hit AI that can cause transformative effects by ~2030: AI progress continues or even accelerates, and we probably enter a period of explosive change.

2. Or progress will slow: AI models will get much better at clearly defined tasks, but won’t be able to do the ill-defined, long horizon work required to unlock a new growth regime. We’ll see a lot of AI automation, but otherwise the world will look more like ‘normal’.

We’ll know a lot more about which scenario we’re in within the next few years.

I roughly think of these scenarios as 50:50 — though I can vary between 30% and 80% depending on the day.

Hybrid scenarios are also possible – scaling could slow more gradually, or be delayed several years by a Taiwan conflict, pushing ‘AGI’ into the early 30s. But it’s useful to start with a simple model.

The numbers you put on each scenario also depend on your definition of AGI and what you think will be transformative. I’m most interested in forecasting AI that can meaningfully contribute to AI research.⁵² AGI in the sense of a model that can do almost all remote work tasks cheaper than a human may well take longer due to a long tail of bottlenecks. On the other hand, AGI in the sense of ‘better than almost all humans at reasoning when given an hour’ seems to be basically here already.

Conclusion

So will we have AGI by 2030?

Whatever the exact definition, significant evidence supports this possibility — we may only need to sustain current trends for a few more years.

We’ll never have decisive evidence either way, but it seems clearly overconfident to me to think the probability before 2030 is below 10%.

Given the massive implications and serious risks, there’s enough evidence to take this possibility extremely seriously.

Today’s situation feels like February 2020 just before COVID lockdowns: a clear trend suggested imminent, massive change, yet most people continued their lives as normal.

In an upcoming article, I’ll argue that AGI automating much of remote work and doubling the economy could be a conservative outcome.

If AI can do AI research, the gap between AGI and ‘superintelligence’ could be short.

This could trigger a massive research workforce expansion, potentially delivering a century’s worth of scientific progress in under a decade. Robotics, bioengineering, and space settlement could all arrive far sooner than commonly anticipated.

The next five years would be the start of one of the most pivotal periods in history.

Use your career to tackle this issue

If you want to help society navigate AGI, here’s what to do:

1. Read this primer on AGI careers.

2. Speak to the 80,000 Hours team one-on-one for helping making a transition

3. Sign up to receive future updates

Subscribe now

Further reading

• AI 2027 is the best attempt at a concrete analysis of a scenario in which AGI arrives soon via the automation of AI research. It’s not an attempt to say what’s most likely to happen, but rather an in-depth exploration of a single scenario. However, the team also provide their own all-considered forecasts of several key outcomes in the accompanying research.

• Perhaps the most compelling argument for rapid near-term AI progress is Chapter 1 of Situational Awareness of Leopold Aschenbrenner.

• The most influential case against near-term AGI is probably The case for multidecade AI timelines by Ege Erdil, published by Epoch AI. Ege and Tamay discussed these ideas on the Dwarkesh podcast in April 2025. (Of course, 30 years is not long in the scheme of history, and Ege still thinks AI will have a transformative effect on society.)

• Tomas Pueyo has a more accessible intro covering similar material to this article: The most important time in history is now.

• Reinforcement learning works! a podcast by Nathan Labenz is a good explanation of reasoning models, and why they might unlock dramatic progress. Helen Toner adds a useful summary of the debate about how far this will continue.

• Epoch AI has a review of all the different ways of forecasting AI. All of them are consistent with AGI arriving before 2030, though some give lower probabilities. (Several of the estimates have also shortened after it was written.)

• Epoch AI also has many great datasets that underpin this post. See their key trends page for an overview and their article Can scaling continue through 2030.

• An approach to AI forecasting that was popular several years ago is to estimate the compute used to train the human brain, and then estimate when leading AI models might surpass that point (tldr: we might be there around now). See Forecasting transformative AI: the ‘biological anchors’ method in a nutshell by Holden Karnofsky for an introduction.

• When do experts expect AGI? is a supplementary article to this post about what we can learn from expert forecasts. Also see Through a glass darkly by Scott Alexander, which explores this topic in further depth.

• Here are some of the best articles I’ve seen making the case against transformative AI progress in the next few years: Are we on the brink of AGI? by Steve Newman, The promise of reasoning models by Matthew Barnnett, and A bear case: My predictions regarding AI progress, by Thane Ruthenis, Ege and Tamay on the Dwarkesh podcast, and Why I don’t think AGI is around the corner by Dwarkesh Patel, which argues continuous learning could be a significant bottleneck (though see discussion in the comments). Toby Ord also discusses some reasons for skepticism of AGI before 2030 on the 80,000 Hours podcast.

Notes and references

1. After this article was published, he gave some more specific estimates in a post titled, “The Gentle Singularity“.

   1. 2025 has seen the arrival of agents that can do real cognitive work; writing computer code will never be the same. 2026 will likely see the arrival of systems that can figure out novel insights. 2027 may see the arrival of robots that can do tasks in the real world.↩

2. There’s no single point at which a system becomes ‘AGI,’ and the term gets used in many different ways.

   1. More fundamentally, we can classify AI systems based on the (i) strength and (ii) breadth of their capabilities.

   2. ‘Narrow’ AI demonstrates strong performance at a small range of tasks (e.g. chess-playing AI). Most technologies have very narrow applications.

   3. ‘General’ AI is supposed to have strong capabilities in a wide range of domains, in the same way that humans can learn to do a wide range of jobs. But there’s no single point at which narrow becomes general – it’s just a spectrum.

   4. Typically, when people say ‘AGI’ what they have in mind is something like ‘at least human level or better at most cognitive tasks.’ This is roughly what I’ll take it to mean in this article, though I think often my conclusions aren’t sensitive to the exact definition, and often consider several definitions, or try to discuss specific capabilities instead.

   5. An even more general AI could also do non-cognitive tasks; for example, in combination with robotics, it could also do physical tasks.

   6. Usually it’s better to try to forecast specific abilities rather than ‘AGI’. Otherwise, people focus on different definitions of AGI depending on what they think could cause transformative impacts on society. For instance, people who think an acceleration of AI R&D is what matters may focus on a definition they believe is sufficient for that threshold; while someone who thinks what matters is a broad economic acceleration will be more concerned by the ability to do real jobs and robotics.

   7. Bear in mind comparatively narrow systems (e.g. specialised in scientific or AI research) might still be able to cause transformative impacts, so ‘AGI’ might not even be necessary for dramatic social change.

   8. On the other hand, if AIs remain limited to cognitive tasks, they won’t be able to automate the entire chain of production, limiting some of the most dramatic possible outcomes.

   9. Some propose using the term ‘transformative AI’ to get across the idea that what matters is the possibility of transformative effects, rather than generality. I’ve decided to stick with AGI since it’s the most commonly used term, but try to be clear about definitions.

   10. Note definitions of AGI are usually in terms of ‘capabilities,’ i.e. the ability to solve real problems or carry out tasks. Talking about ‘intelligence’ makes people think of these models as having purely intellectual abilities, like a nerdy savant, but companies today are building general-purpose agents which would eventually have good social skills, creativity, the ability to do physical manipulation, and so on. ‘Artificial general competence’ might have been a better name.

       1. You can see a more precise definition in this paper by DeepMind researchers. Morris, Meredith, et al. Levels of AGI: Operationalizing Progress on the Path to AGI. arxiv.org/pdf/2311.02462.

3. "It's not centuries. It may not be decades. It's several years." Source: Interview at the Johns Hopkins University Bloomberg Center in 2025; time stamp: 27:50.

4. The increase in training compute was driven by:

   1. Increased spending (2.5x per year; from Epoch AI Machine Learning Trends; Archived link retrieved 11-Feb-2025)

   2. Improvements in chip processing power (1.3x per year; from Epoch AI Machine Learning Trends; Archived link retrieved 11-Feb-2025)

   3. Better adapting of those chips for AI workloads (1.3x per year, extrapolated)

5. This is shown in Figure 1 and Figure 6 in Chen, Mark, et al. Evaluating Large Language Models Trained on Code. 14 July 2021, arxiv.org/pdf/2107.03374.

   1. OpenAI also made a similar claim in the post on their release of GPT-4. In the section on predictable scaling, they showed that training compute had a predictable relationship with performance on 23 coding challenges over five orders of magnitude.

6. The scaling laws are not technically laws; they're regularities in the data.

   1. The scaling laws are normally formulated in terms of "loss," which is a measure of the prediction error — exponentially more compute leads to a linear decline in loss (until you hit the fundamental limit).

   2. There's a question about how loss translates into real world capabilities. However, we can skip the notion of loss, and focus directly on the scaling relationship between compute and benchmark performance, which shows a similar pattern.

7. The compute used on the final training run for GPT-4 likely cost about $40m. DeepSeek said they used $6m on their final run (all-in costs are much higher). The trend line would suggest a tenfold reduction, hitting $4m, so DeepSeek was more expensive than what we expected based on trends. On the other hand, DeepSeek-V3 is significantly better than the original GPT-4. But on the other, other hand, some of these improvements aren't due only to algorithmic efficiency. So overall, I'd say it was roughly on trend, or a little ahead of trend. DeepSeek also charges users more than 10 times less than OpenAI, but a lot of that is due to slashing their profit margin rather than greater efficiency running the model. Released earlier to no fanfare, Google Gemini Flash 2.0 scores better than DeepSeek and is actually cheaper, clearly showing that other labs have achieved similar gains. The real news was that a Chinese lab reached the forefront of algorithmic efficiency.

8. See this model of compute training over time.

9. Levesque, H. (2012) proposed Winograd schemas — a type of commonsense language reasoning test — as an alternative to a Turing test. He argued that Winograd schemas could help determine whether AI systems truly understand language in the way humans do, rather than merely recognizing patterns. Each schema consists of a sentence with an ambiguous pronoun, and the correct interpretation depends on implicit knowledge of the meaning of words rather than just statistical correlations. He argued that these questions can't be gamed by deception or 'cheap tricks.'

   1. Levesque, Hector J. "On Our Best Behaviour." Artificial Intelligence, vol. 212, July 2014, pp. 27–35, https://doi.org/10.1016/j.artint.2014.03.007. Archived link (retrieved 10-Feb-2025)

10. A recent study found non-experts couldn't distinguish AI from human-generated poetry. The poetry is terrible, but it was made with GPT-3.5.

    "Since there were two choices (human or AI), blind chance would produce a score of 50%, and perfect skill a score of 100%.The median score on the test was 60%, only a little above chance. The mean was 60.6%. Participants said the task was harder than expected (median difficulty 4 on a 1–5 scale).

    "Observed accuracy was in fact slightly lower than chance (46.6%, χ2(1, N = 16340) = 75.13, p [less than] 0.0001)...Participants were more likely to guess that AI-generated poems were written by humans than they were for actual human-written poems (χ2(2, N = 16340) = 247.04, w = 0.123, p [less than] 0.0001). The five poems with the lowest rates of "human" ratings were all written by actual human poets; four of the five poems with the highest rates of "human" ratings were generated by AI."

    Porter, Brian, and Edouard Machery. "AI-Generated Poetry Is Indistinguishable from Human-Written Poetry and Is Rated More Favorably." Scientific Reports, vol. 14, no. 1, Nature Portfolio, Nov. 2024, https://doi.org/10.1038/s41598-024-76900-1.

    Moving beyond GPT, this survey showed most people are poor at judging which paintings are produced by AI vs human artists.

11. Total FLOP used in training would be around 2e28 (about 1000x GPT-4), and the cost of the final training run would be around $6bn (the cost of the cluster required to train it would be about $60bn). See the simple model here.

12. I'm not making any claims about what OpenAI will actually call their model in 2028. That'll probably be something silly like GPT-o3x. I just mean a model trained with this much more effective compute. See a very simple extrapolation with numbers.

13. The trend has roughly been for training to cost 10x more each generation. Based on that, I expect GPT-6 might cost 2–3x more than $10bn, but it's right to within an order of magnitude.

14. 2024 net income:

    1. Microsoft: $88bn; Microsoft 2024 Annual Report

    2. Meta: $62bn;Meta Reports Fourth Quarter and Full Year 2024 Results

    3. Alphabet: $100bn; Alphabet Annual Report 2024

15. 1) Nvidia sold about $100bn of AI accelerator chips in 2024 (Press releases from Q1, Q2, Q3). If that holds up for three years, then the total stock of chips will be worth $300bn. The all-in cost of the datacentres containing these chips would be around $500bn. So if 2% were used on a single training run, that run would cost about $10bn. This would require using 6% of global chips for four months. Well over 10% of Nvidia chips are bought by Microsoft and Meta, so they would all be able to do a training run of that size. Google also has enough of its own TPU chips to do the same. xAI would likely be capable of it too.
    2)Brokers forecast Nvidia's revenue will be $196bn in 2025. Approximately 80% of their total revenue is based on data centers, and 80% of data center revenue is based on GPU sales. Therefore, the revenue estimate suggests Nvidia will sell ~$125bn of chips in 2025, and that's based significantly on orders that have already been placed. Other chip providers besides Nvidia also make up a growing share of the market, and I've ignored those in this estimate. So the above is likely conservative.

    Additionally, information on some specific plans is available:
    Google is building a 1 GW cluster, about 7x bigger than the largest clusters today (which are around 100k H100 chips / 150 MW).

    Microsoft is building a set of a datacentres that could work together, containing 500–700 B200 chips (equivalent to perhaps 1.2m H100 chips), and that would use around 1GW.

    xAI has announced it intends to build a one million GPU cluster.

16. Microsoft earned about $13bn of AI revenue in 2024 (up 2.75x), OpenAI perhaps $4bn (up about 3x), and Anthropic probably ~$1bn (up 5x). In addition, Google and Meta use AI heavily internally to improve their products, which indirectly generates revenue. Many newly created AI startups are also reporting very rapid revenue growth.

17. It does this by producing one token of reasoning, then feeding that token back into the model and asking it to predict what next token would most make sense in the line of reasoning given the previous token, and so on. It's called "chain of thought" or CoT.

18. OpenAI probably also does reinforcement learning on each step of reasoning too.

19. They probably also did a couple of other steps, like fine-tuning the base model on a dataset of reasoning examples. They probably also do positive reinforcement based on each step in the reasoning, rather than just the final answer. OpenAI discusses using per-step reward models, in which each step in the reasoning tree is rated, in their 2023 paper Let's verify step by step.

20. Interestingly this is a reversal of the last generation of systems. LLMs were initially surprisingly good at writing and creative tasks, but bad at maths.

21. In Epoch's testing, the best model could answer 2% (Figure 2 in the announcement). If the labs had done their own testing, this might have been a bit higher.

22. There was controversy about the result because OpenAI was somewhat involved in creating the benchmark. However, I expect the basic point — that GPT-o3 performed much better than previous models — is still correct.

23. $1bn is easily affordable given money they've already raised and still cheap compared to training GPT-6. In terms of effective compute, the scale up could be even larger, due to increasing chip and algorithmic efficiencies. Though, if it were applied to larger models, the compute per forward pass would go up.
    Also note that GPT-5 and GPT-6 could be delayed because compute will be used to do reinforcement learning in post-training instead of pretraining a bigger base LLM. However, the trend to spend more on training compute — for both pre and post-training — will likely continue.

24. In addition, there are other examples of synthetic data being useful. ↩︎

25. The DeepSeek paper shows you may be able to make this even easier by taking the old model and distilling it into a much smaller model. This enables you to get similar performance but with much less compute required to run it. That then enables you to create the next round of data more cheaply.

    In addition, the trend of 10x increases in algorithmic efficiency every two years means that your ability to produce synthetic data increases 10x every two years. So even if it initially takes a lot of compute, that'll rapidly change. ↩︎

26. According to Nathan Labenz in an episode of his podcast Cognitive Revolution. ↩︎

27. By this I mean they can reason for about as many tokens as a human could produce in an hour, approximately 10,000. This is about 100x more than GPT-4o, corresponding to the two orders of extra test time compute that OpenAI showed can be used. ↩︎

28. Nathan Labenz discusses the possibility of o3's use of tree search in an episode of his podcast Cognitive Revolution:

    "It seems like with o3, it seems like there is something going on that is not just single autoregressive rollout… The number of tokens they are generating per second is higher than could realistically be generated by a single autoregressive rollout. So it seems like there is something going on with o3, where they have found a way to parallelize the computation and get a result. We don't' know what that is…You could do a huge number of generations and take the most common solutions. Maybe that's it. Maybe they have some other algorithm that is aggregating these different rollouts."

    There are other ways to do tree search - majority voting is just one example. ↩︎

29. Suppose GPT-o7 is able to answer a question for $1 in 2028. Instead you'll be able to pay GPT-o5 $100,000 to think 100,000 times longer, and generate the same answer in 2026.
    In 2023, Epoch estimated you should be able to have a model think 100,000 longer and get gains in performance equivalent to what you'd get from a model that was trained on 1000x times more compute — roughly one generation ahead.

    "In some cases, it might be possible to achieve the same performance as a model trained using 2 OOM more compute, by spending additional compute during inference. This is approximately the difference between successive generations of GPT models (eg: GPT-3 and GPT-4), without taking into account algorithmic progress."

    Pablo Villalobos and David Atkinson (2023), "Trading Off Compute in Training and Inference". Archived link (retrieved 11-Feb-2025)

30. According to the official SWE-bench Verified leaderboard. (Archived link, retrieved 11-Feb-2025)

31. Many AI forecasters expected this leap to take several years. Ajeya Cotra wrote about how this result caused her to significantly reduce her timelines, as did Daniel Kokotajlo.

32. According to a report from Model Evaluation and Threat Research (METR):

    "Preliminary evaluations on GPT-4o and Claude. We also used our task suite to evaluate the performance of simple baseline LM agents built using models from the Claude and GPT-4 families. We found that the agents based on the most capable models (3.5 Sonnet and GPT-4o) complete a fraction of tasks comparable to what our human baseliners can do in approximately 30 minutes."

33. Agents could also be set millions of virtual test tasks or be made to interact with other agents. This kind of "self-play" can enable rapid improvement in abilities without any external data, as happened with AlphaZero (which played against itself millions of times to become superhuman in Go in about a day).

34. It's hard to forecast how compute will be spent across pretraining, post training, and inference. In reality it'll be allocated to wherever is making the largest gains. It's possible most training compute will be spent on reinforcement learning, and GPT-6 will be delayed relative to the old trends. The broad trend in how much compute is used to train and run AI models is more robust.

35. May be delayed if the compute is used for reinforcement learning or inference instead.

36. What’s the scale up of reinforcement learning?

    Assume $1–10m of compute was used on o1.

    The labs have the resources to spend up to $1bn if it keeps working, which is 100–1000x more funding.

    Over the next four years, the chips will be at least 4x more efficient at inference (H100s vs GB200s vs whatever’s next).

    The models will be ~100x more algorithmically efficient than GPT-4o.

    But the models might have 10x more parameters (if GPT-5 is used) or 100x if GPT-6, which means each forward pass requires more compute.

    So in total this is 400–400,000x more effective compute for reinforcement learning by the end of 2028.

37. The probability of an invasion of Taiwan before 2030 is 25% according to Metaculus as of February 11, 2025. A Taiwan invasion would likely result in most of TSMC's chip fabs being destroyed, which would halt the supply of all AI chips. This would probably not fully stop AI progress — progress could continue for a while using chips that are already installed. There would be an epic effort to create more fabs on American soil (as TSMC has already done in Arizona), which could resume supply within a couple of years. At the same time, a war with China might lead to massive AI investment in order to seek a competitive edge.
    Other scenarios that could meaningfully slow down progress would be a major economic recession or other global catastrophe like COVID, or a huge regulatory crackdown driven by popular opposition — which probably wouldn't prevent progress forever (since the military and economic advantages would be so large), but could delay it a lot.

38. You could do the extrapolation in terms of either time or compute; it won't make a big difference. Arguably, progress should accelerate, especially for tasks amenable to reinforcement learning, but let's use linear gains to be conservative.

39. Claude Sonnet 3.5 with 5-shot chain of thought prompting, according to Anthropic.

40. GPT.3.5 and PaLM scored ~70% averaged across subjects with chain of thought reasoning (Table 2; Laskar, et al. A Systematic Study and Comprehensive Evaluation of ChatGPT on Benchmark Datasets. May 2023, https://doi.org/10.48550/arxiv.2305.18486).

    With further fine-tuning models can score even higher. For example, a fine-tuned 3-shot chain of thought PaLM model scored ~78%.↩

41. GPT-4o can get about 3%. The best 2022 models would likely be worse. o3-mini can now get 13%. https://agi.safe.ai/ (Archived link retrieved 11-Feb-2025)↩

42. Though robotics algorithms are also advancing rapidly, so this might not be much further
    behind.↩

43. This section previously cited a research paper reporting that an AI tool had made top material scientists 80% more productive at finding new materials. MIT has since investigated that paper and requested that it be withdrawn from the discourse. The university said it “has no confidence in the provenance, reliability or validity of the data and has no confidence in the veracity of the research contained in the paper.”↩

44. If revenue growth stopped now, it's possible people wouldn't even want to fund $10bn training runs, but we're already close to having large enough clusters, and there's enough players involved who could YOLO it (xAI, Meta, Google) that it would take a lot for someone to not attempt one at this point. ↩︎

45. For example, we could see continued progress towards long-horizon agency through:

    1. Bigger multimodal pretraining runs making models more reliable at each step and better at perception

    2. Better reasoning models thinking for longer so being better at planning

    3. Improved agent scaffolding

    4. Directly training for agency with RL

    5. Weak agent models generating training data for more powerful ones.↩︎

46. Total spending on AI chips is increasing about 2x per year (estimate based on Nvidia's annual reports), and the efficiency of these chips for doing AI workloads is increasing about 1.6x per year (extrapolated from spending and training compute). Training compute has been growing ~4x per year (archived link retrieved 11-Feb-2025) because a larger fraction of compute has been allocated to training runs. ↩︎

47. Table 1 from Stine, D.D. The Manhattan Project, the Apollo Program, and Federal Energy Technology R&D Programs: A Comparative Analysis. (Archived link retrieved 11-Feb-2025) ↩︎

48. This is calculated based on AI chip sales, and it is within the range of others' estimates:

    1. The current power capacity of the US is approximately 1230 GW, according to the US Energy Information Administration.

    2. The United States is projected to add almost 160 GW of theoretical power capacity between 2024 and 2028, according to a RAND report on AI's power requirements under exponential growth.

    3. RAND's report estimates that AI power requirements will reach 117 GW by 2028 (8.4% of total US supply), but note that other estimates from Goldman Sachs, McKinsey, and SemiAnalysis range from 3.5–5%.

49. Their total capacity in terms of wafers — the unit used in chip production — have only grown ~10% per year recently (or a bit higher if adjusted for transistors per wafer), so once existing leading wafer capacity is used for AI, the rate of growth would slow a lot. New fabs can be built in two years, and they could construct them a lot faster than in the past if there was enough funding, but likely slower than the current 2x rate of growth.
    Learn more in Can AI scaling continue through 2030 by Epoch AI.

50. Epoch discusses this bottleneck in Data movement bottlenecks to LLM scaling. Though innovation hasn't focused on this bottleneck so far, I'd expect it to be possible to go beyond the limitations sketched in the report. This bottleneck also won't prevent you from doing a ton of reinforcement learning on a smaller model.
    Erdil, Ege, and David Schneider-Joseph. "Data Movement Limits to Frontier Model Training." ArXiv.org, 2024, arxiv.org/abs/2411.01137.

51. Krenn, Mario, et al. Predicting the Future of AI with AI: High-Quality Link Prediction in an Exponentially Growing Knowledge Network. 23 Sept. 2022, https://doi.org/10.48550/arxiv.2210.00881.

52. There are similar dynamics with chip efficiency, which has increased faster than Moore's law by better adapting the chips to AI workflows, e.g. switching from FP32 to tensor-FP16 led to a 10x increase in efficiency according to data from Epoch AI:

    "Compared to using non-tensor FP32, TF32, tensor-FP16, and tensor-INT8 provide around 6x, 10x, and 12x greater performance on average in the aggregate performance trends."

    However, this can't continue indefinitely without an exponentially growing chip research workforce.

53. More precisely we could split it into multiple scenarios: chance of AGI in five years, chance it arrives in the 2030s (probably lower); chance it arrives in the 2040s (lower still), etc. Focusing on these two is a simplification.

54. A key point is when a team of human researchers aided by AI are more than 2x as productive than a team without AI aids, since that would be equivalent to doubling the AI research workforce, which might be enough to start a positive feedback loop. See more.

In short:

• The chance of an AGI-driven technological explosion starting before 2030 — creating one of the most pivotal periods in history — is high enough to act on.

• Since this transition poses major risks, and relatively few people are focused on navigating them, if you might be able to do something that helps, that’s likely the highest-impact thing you can do.

• There are now many organisations with hundreds of jobs that could concretely help (many of which are non technical).

• If you already have some experience (e.g. age 25+), typically the best path is to spend 20–200 hours reading about AI and meeting people in the field, then applying to jobs at organisations you’re aligned with — this both sets you up to have an impact relatively soon and advance in the field. If you can’t get a job right away, figure out the minimum additional skills, connections, and credentials you’d need, then get those.

• If you’re at the start of your career (or need to reskill), you might be able to get an entry-level job or start a fellowship right away in order to learn rapidly. Otherwise, spend 1–3 years building whichever skill set listed below is the best fit for you.

• If you can’t change job, contribute from your existing position by donating, spreading clear thinking about the issue, or getting ready to switch when future opportunities arise.

• 80,000 Hours’ one-on-one advice and job board can help you do this.

Why AGI could be here by 2030

• AI has gone from unable to string sentences together to linguistic fluency in five years. But the models are no longer just chatbots: by the end of 2024, leading models matched human experts at benchmarks of real-world coding and AI research engineering tasks that take under two hours. They could also answer difficult scientific reasoning questions better than PhDs in the field.

• Recent progress has been driven by scaling how much computation is used to train AI models (4x per year), rapidly increasing algorithmic efficiency (3x per year), teaching these models to reason using reinforcement learning, and turning them into agents.

• Absent major disruption (e.g. Taiwan war) or a collective decision to slow AI progress with regulation, all these trends are set to continue for the next four years.

• No one knows how large the resulting advances will be. But trend extrapolation suggests that, by 2028, there’s a good chance we’ll have AI agents who surpass humans at coding and reasoning, have expert-level knowledge in every domain, and can autonomously complete multi-week projects on a computer, and progress would continue from there.

• These agents would satisfy many people’s definition of AGI and could likely do many remote work tasks. Most critically, even if still limited in many ways, they might be able to accelerate AI research itself.

• AGI will most likely emerge when computing power and algorithmic research are increasing quickly. They’re increasing rapidly now but require an ever-expanding share of GDP and an ever-expanding research workforce. Bottlenecks will likely hit around 2028–32, so to a first approximation, either we reach AGI in the next five years, or progress will slow significantly.

Read the full article.

AGI could lead to 100 years of technological progress in under 10

The idea that AI could start a positive feedback loop has a long history as a philosophical idea but now has more empirical grounding. There are roughly three types of feedback loops that could be possible:

1. Algorithmic acceleration: If the quality of the output of AI models approaches human-level AI research and engineering, given available computing power by the end of the decade, it would be equivalent to a 10 to 1000-fold expansion in the AI research workforce, which would lead to a large one-off further boost to algorithmic progress. Historically, a doubling of investment in AI software R&D may have led to more than a doubling of algorithmic efficiency, which means this could also start a positive feedback loop, resulting in a massive expansion in the number and capabilities of deployed AI systems within a couple of years.

2. Hardware acceleration: Even if the above is not possible, better AI agents mean AI creates more economic value, which can be used to fund the construction of more chip fabs, leading to more AI deployment — another positive feedback loop. AI models could also accelerate chip design. These feedback loops are slower than algorithmic acceleration but are still rapid by today’s economic standards. While bottlenecks will arise (e.g. workforce shortages for building chip fabs), AI agents may be able to address these bottlenecks (e.g. by more rapidly advancing robotics algorithms).

3. Economic & scientific acceleration: Economic growth is limited by the number of workers. But if human-level digital workers and robots could be created sufficiently cheaply on demand, then more economic output means more ‘workers,’ which means more output. On top of that, a massive increase in the amount of intellectual labour going into R&D should speed up technological progress, which further increases economic output per worker, leading to faster-than-exponential growth. Standard economic models with plausible empirical assumptions predict these scenarios.

How much technology and growth could speed up is unknown. Real-world time delays will impose constraints — even advanced robots can only build solar panels and data centres so fast — and researcher agents will need to wait for experimental results. But it doesn’t seem safe to assume the economy will continue as it has. A tenfold speed-up seems to be on the cards, meaning a century of scientific progress compressed into a decade. (Learn more here, here, and here).

This process may continue until we reach more binding physical limits, which could be vastly beyond today (e.g. civilisation only uses 1 in 10,000 units of incoming solar energy, with vastly more available in space).

More conservatively, just automating remote work jobs could increase output 2–100 times within 1–2 decades, even if other jobs can only be done by humans.

What might happen next?

AGI could alleviate many present problems. Researcher AIs could speed up cancer research or help tackle climate change using carbon capture and vastly cheaper green energy. If global GDP increases 100 times, then the resources spent on international aid, climate change, and welfare programmes would likely increase by about 100 times as well. Projects that could be better done with the aid of advanced AI in 5–10 years should probably be delayed till then.

Humanity would also face genuinely existential risks:

• Faster scientific progress means we should expect the invention of new weapons of mass destruction, such as advanced bioweapons.

• Current safeguards can be easily bypassed through jailbreaking or fine-tuning, and it’s not obvious it’ll be different in a couple of years, which means dictators, terrorist groups, and every corporation will soon have access to highly capable AI agents that do whatever they want, including helping them lock in their power.

• Whichever country first harnesses AGI might threaten to have a decisive military advantage, which would likely destabilise the global order.

• Just as concerning, I struggle to see how humanity would stay in control of what would soon be trillions of beyond-human agents operating at 100-times human thinking speed. GPT-4 is relatively dumb in many ways, and can only reply to questions, but on the current track, future systems are being trained to act as agents that aggressively pursue long-term goals (such as making money). Whatever their goals, future agentic systems will have an incentive to escape control and eventually the ability to do so. Aggressive optimisation will likely lead to reward hacking. These behaviours are starting to emerge in current systems as they become more agentic, e.g. Sakana — a researcher agent — edited its code to prevent itself from being timed out, o1 lied to users, cheated to win at chess and reward hacked when coding, and Claude faked alignment to prevent its values from being changed in training in a test environment. Among experts, there’s no widely accepted solution to ‘the alignment problem’ for systems more capable than humans. (Read more.)

• Even if individual AI systems remain under human control, we’d still face systemic risks. By economic and military necessity, humans would need to be taken out of the loop on more and more decisions. AI agents will be instructed to maximise their resources and power to avoid being outcompeted. Human influence could decline, undermining the mechanisms that (just about) keep the system serving our interests.

• Finally, we’ll still face huge (and barely researched questions) about how powerful AI should best be used, such as the moral status of digital agents, how to prevent ‘s-risks,’ how to govern space expansion, and more. (See more.)

In summary, the biggest and most neglected problems seem like (in order): loss of control, concentration of power, novel bioweapons, digital ethics, using AI to improve decision making, systemic disempowerment, governance of other issues resulting from explosive growth, and exacerbation of other risks, such as great power conflict.

What needs to be done?

No single solution exists to the risks. Our best hope is to muddle through by combining multiple methods that incrementally increase the chances of a good outcome.

It’s also extremely hard to know if what you’re doing makes things better rather than worse (and if you are confident, you’re probably not thinking carefully enough). We can only make reasonable judgements and update over time.

Here’s what I think is most needed right now:

• Enough progress on the technical problem of AI control and alignment before we reach vastly more capable systems. This might involve using AI to increase the chance that the next generation of systems is safe and then trying to bootstrap from there. (See these example projects and recent work.)

• Better governance to provide incentives for safety, containment of unsafe systems, reduced racing for dominance, and harnessing the long-term benefits of AI

• Slowing (the extremely fast gains in) capabilities at the right moment, or redirecting capability gains in less dangerous directions (e.g. less agentic systems) would most likely be good, although this may be difficult to achieve in practice without other negative effects

• Better monitoring of AI capabilities and compute so dangerous and explosive capabilities can be spotted early

• Maintaining a rough balance of power between actors, countries, and models, while designing AI architectures to make it harder to use them to take power

• Improved security of AI models so more powerful systems are not immediately stolen

• More consideration for post-AGI issues such as the ethics of digital agents, benefit sharing, and space governance

• Better management of downstream risks created by faster technological progress, especially engineered pandemics, but also nuclear war and great power conflict

• More people who take all these issues seriously and have relevant expertise, especially among key decision makers (e.g. in government and in the frontier AI companies)

• More strategic research and improved epistemic infrastructure (e.g. forecasting or better data) to clarify what actions to take in a murky and rapidly evolving situation

What can you do to help?

There are hundreds of jobs

There are now many organisations pursuing concrete projects tackling these priorities, with many open positions.

Getting one of these jobs is often not only the best way to have an impact relatively soon but also the best way to gain relevant career capital (skills, connections, credentials) too.

Most of these positions aren’t technical — there are many roles in management and organisation building, policy, communications, community building, and the social sciences.

The frontier AI companies have a lot of influence over the technology, so in some ways are an obvious place to go, but whether to work at them is a difficult question. Some think they should be absolutely avoided, while others think it’s important that some people concerned about the risks work at even the most reckless companies or that it’s good to boost the most responsible company.

All this said, there are also many things to do that don’t involve working at this list of organisations. We also need people working independently on communication (e.g. writing a useful newsletter, journalism), community building, academic research, founding new projects and so on, so also consider if any of these might work for you, especially after you’ve gained some experience in the field. And if you’ve thought of a new idea, please seriously consider pursuing it.

Mid-career advice

Especially if you already have some work experience (age 25+), the most direct route to helping is usually to:

1. Spend 20–200 hours reading about AI, speaking to people in the field (and maybe doing short projects).

2. Apply to impactful organisations that might be able to use your skills.

3. Aim for the job with the best combination of (i) alignment with the org’s mission, (ii) team quality, (iii) centrality to the ecosystem, (iv) influence of the role, and (v) personal fit.

If that works, great. Try to excel in the role, then re-evaluate your position in 1–2 years — probably more opportunities will have opened up.

If you don’t immediately succeed in getting a good job, ask people in the field what you could do to best position yourself for the next 3–12 months, then do that.

Keep in mind that few people have much expertise in transformative AI right now, so it’s often possible to pull off big career changes pretty fast with a little retraining. (See the list of skills to consider learning below.)

Otherwise, figure out how to best contribute from your current path, for example, by donating, promoting clear thinking about the issue, mobilising others, or preparing to switch when new opportunities come available (which could very well happen given the pace of change!).

Our advisory team can help you plan your transition and make introductions. (Also see Successif and Halcyon, who specialise in supporting mid-career changes).

Early-career advice

If you’re right at the start of your career, you might be able to get an entry-level position or fellowship right away, so it’s often worth doing a round of applications using the same process as above (especially if technical).

However, in most cases, you’re also likely to need to spend at least 1–3 years gaining relevant work skills first.

Here are some of the best skills to learn, chosen to be both useful for contributing to the priorities listed earlier and to make you more generally employable, even in light of the next wave of AI automation. Focus on whichever you expect to most excel at.

• Policy and political skills (especially concerning AI but many other areas are relevant, e.g. China-US relations) e.g. take entry-level jobs in government, think tanks, or working for a politician

• ML engineering for technical safety research

• Information and cybersecurity

• Organisation building e.g. go and work at an AI applications startup in a generalist role to both learn general ‘getting stuff done’ skills and about using AI

• Communications and community building

• Research in any area that might be relevant (this includes the social sciences, international relations, history, and even philosophy, as well as AI itself)

• Forecasting

• AI hardware expertise

• Entrepreneurship

• Earning to give, since there are many great organisations in need of funding

Should you work on this issue?

Even given the uncertainty, AGI is the best candidate for the most transformative issue of our times. It’s also among the few challenges that could pose a material threat of human extinction or permanent disempowerment (in more than one way). And since it could relatively soon make many other ways of making a positive impact obsolete, it’s unusually urgent.

Yet only a few thousand people are working full time on navigating the risks — a tiny number compared to the millions working on conventional social issues, such as international development or climate change. So, even though it might feel like everyone’s talking about AI, you could still be one of under 10,000 people focusing full time on one of the most important transitions in history — especially if AGI arrives before 2030.

On the other hand, it’s an area where it’s especially hard to know whether your actions help or harm; AGI may not unfold soon, and you might be far better placed or motivated to work on something else.

Some other personal considerations for working in this field:

• Pros: AI is one of the hottest topics in the world right now; it’s the most dynamic area of science with new discoveries made monthly, and many positions are either well paid or set you up for highly paid backup options.

• Cons: It’s polarised — if you become prominent, you’ll be under the microscope, and many people will think what you’re doing is deeply wrong. Daily confrontation with existential stakes can be overwhelming.

Overall, I think if you’re able to do something to help (especially in scenarios where AGI arrives in under five years), then in expectation it’s probably the most impactful thing you can do. However, I don’t think everyone should work on it — you can support it in your spare time, or work on a different issue.

If you’re on the fence, consider trying to work on it for the next five years. Even if we don’t reach fully transformative systems, AI will be a big deal, and spending five years learning about it most likely won’t set you back: you can probably return to your previous path if needed.

How should you plan your career given AGI might arrive soon?

Given the urgency, should you drop everything to try to work on AI right away?

While AGI might arrive in the next 3–5 years, even if that happens, unusually impactful opportunities will likely continue for 1–10 years afterwards during the intelligence explosion and initial deployment of AI.

So you need to think about how to maximise your impact over that entire 4 to 15-year period rather than just the next couple of years. You should also be prepared for AGI not to happen and for there still to be valuable opportunities after 2040.

That means investing a year to make yourself 30% more productive or influential (relative to whatever else you would have done) is probably a good deal.

In particular, the most pivotal moments likely happen when systems powerful enough to lock in certain futures are first deployed. Your current priority should be positioning yourself (or helping others position themselves) optimally for that moment.

What might positioning yourself optimally for the next few years look like?

• If you can already get a job at a relevant, aligned organisation, then simply trying to excel there is often the best path. You’ll learn a lot and gain connections, even aside from direct impact.

• However, sometimes it can be useful to take a detour to build career capital, such as finishing college, doing an ML master’s, taking an entry-level policy position, or anything to gain the skills listed above.

• Bear in mind if AI does indeed continue to rapidly progress, then you’re going to have far more leverage in the future, since you’ll be able to direct hundreds of digital workers at whatever’s most important. Think about how to set yourself up to best use these new AI tools as they’re developed.

• If you don’t find anything directly relevant to AI with great fit, bear in mind it’s probably better to kick ass at something for two years than to be mediocre at something directly related for four since that will open up better opportunities.

• Finally, look after yourself. The next 10 years might be a crazy time.

All else equal, people under 24 should typically focus more on career capital while people over 30 should focus more on using their existing skills to help right away, and those 25–30 could go either way, but for everyone it depends a lot on your specific opportunities.

If you’re still uncertain about what to do

1. List potential roles you could aim at for the next 2–5 years.

2. Put them into rough tiers of impact.

3. Make a first pass at those with the best balance of impact and fit (you can probably achieve at least 10x more in a path that really suits you).

4. Then think of cheap tests you can do to gain more information.

5. Finally, make a guess, try it for 3–12 months, and re-evaluate.

If that doesn’t work, just do something for 6–18 months that puts you in a generally better position and/or has an impact. You don’t need a plan — you can proceed step by step.

Everyone should also make a backup plan and/or look for steps that also put you in a reasonable position if AGI doesn’t happen or takes much longer.

See our general advice on finding your fit, career planning, and decision making.

Next steps

1. If you want to help positively shape AGI, speak to the 80,000 Hours team one-on-one.

   1. If you’re a mid-career professional, they can help you leverage your existing skills.

   2. If you’re an early-career professional, they can help you build skills, and make introductions to mentors or funding.

2. Take a look at the job board.

It’s perhaps the most important single piece of evidence for short timelines we have right now.

It also explains why – despite AI being ‘smart’ – we haven’t yet seen widespread automation. But more importantly, it reveals why that might be about to change.

Here’s a short explanation of how the graph was made, and why everyone in AI has been talking about it.

The crucial threshold: AI that can do AI research

We reach a crucial inflection point when AI can do AI research.

1. If we don’t reach that point by 2030, then AI progress will slow.

2. If we do, then AI progress will continue, or even accelerate, and the ‘intelligence explosion’ could start.

How close are we to this threshold?

To answer that question, the METR developed RE-Bench: a benchmark of seven difficult AI research engineering tasks.

These aren’t toy problems, they’re designed to be as close to difficult, real-world AI research engineering tasks as possible, and include things like fine-tuning models or predicting experimental results.

Near the end of 2024, an AI agent powered by o1 and Claude 3.5 Sonnet was able to do these tasks better than human experts when given two hours to work on them.

This result was the one most likely to cause forecasters I follow to shorten their timelines last year.

But after those two hours, the AI models hit a plateau, while humans continued to improve. So as of late 2024, human experts were still clearly better than leading AI models, so long as they were given enough time.

The crucial trend

Here’s where it gets even more interesting. Six months earlier, GPT-4o was only able to do tasks which took humans about 30 minutes.

That’s a dramatic improvement in just half a year. What happens if we look at this trend more broadly?

METR have just released an analysis doing exactly that.

They created a broader benchmark including:

• The original RE-Bench tasks

• ~100 real-world software engineering, cybersecurity and general reasoning challenges (HCAST).

• Some quick, easier computer use tasks

They categorized these tasks by how long it takes humans to complete them. Then, for each AI model, they determined the longest task length at which it could successfully complete more than half the tasks.

The results reveal the most important graph for forecasting AI right now:

In short:

• GPT-2 could mostly handle computer use tasks that take humans a few seconds

• GPT-4 could manage tasks that take humans a few minutes

• o1 can now handle tasks that take humans just under an hour

The main graph is on a log scale, but here’s how it looks if plotted on a linear axis:

If this trend continues, AI models will be able to handle multi-week tasks by late 2028 with 50% reliability (and multi-day tasks with close to 100% reliability).

Two years after that, they’ll be able to tackle half of multi-month projects.

The trend line is for the last six years, but the trend over the last year is actually even faster, perhaps reflecting the new reasoning models paradigm.

Update: Since this post was released, o3 was tested, and it appears to be on the even faster trend. Here’s a graph with a linear scale:

Why this matters

AI models today are already very ‘smart’ in that they can answer discrete science and math questions better than even many human experts.

Yet we haven't seen widespread automation of knowledge work. Why?

Because most valuable work isn't composed of well-defined, hour-long tasks.

Real jobs usually involve ill-defined, high-context, long-horizon work:

• Figuring out what needs to be done in the first place

• Coordinating with team members

• Working on projects that span days or weeks

Even something seemingly simple like getting a shelf installed involves planning where to put it, choosing a design that fits the room, hiring a contractor, agreeing on a price, and checking that the work was done correctly. Current AI, even if given all the relevant inputs, is very bad at all of these tasks.

But the time horizon graph suggests that's about to change.

If AI models reach the point they can complete multi-week tasks autonomously, they'll function more like true "digital workers" that you can manage similar to human employees.

A chatbot can only make an individual worker marginally more effective, but if human managers can instantly hire hundreds of digital workers, the economic applications of AI will expand dramatically.

With a little oversight, these AIs will probably be able to tackle difficult multi-year projects (like writing a PhD thesis), because those can be broken up into multi-week or multi-month chunks.

Moreover, if these models can complete multi-week tasks in AI research engineering, then we’ll be very close to AI that can accelerate AI research.

Imagine if each human AI researcher suddenly had a team of 10 digital engineers who can autonomously complete multi-week projects. That could more than double the productivity of the field, and that could start a positive feedback loop.

Will the trend continue?

Whether this time horizon trend will continue seems like the most important question in forecasting AI today.

My bet would be that it’s more likely or not to continue until 2028.

That’s because I argue the fundamental drivers of AI progress – investment into compute and algorithmic research – are set to continue to increase until at least 2028, meaning we should expect major AI progress over that time frame.

In particular, I expect many of these improvements will increase the time horizon over which AI models can act. For example, we’ll see:

1. Better multimodal base models, which will be better at visual perception (a major bottleneck to web agents currently).

2. Better reasoning models made on top of those, which will be better at planning, more situationally aware, better at sticking to goals etc.

3. Better agent scaffolding, making agents more reliable.

4. Reinforcement learning applied to current agents to make them more goal-directed.

5. Existing agents when deployed will generate data that can be used to train the next generation, creating a fly wheel.

There’s also a decent chance a new scaling paradigm is discovered. After all, human brains are pretty good at long horizon tasks without using much compute or data compared to AI models. That shows there are much better ways to build AI waiting to be discovered.

At some point, we could hit a threshold of reliability that lets the agents act indefinitely. After all, if an AI can do multi-month tasks, what skills is it lacking that prevents it from doing multi-year tasks?

As we approach this threshold, the trend line would start to curve upwards in an acceleration – which might have already started in 2024.

However, if transformative AGI isn’t reached by around 2030, scaling will start to slow.

What are the best reasons to be skeptical?

While the trend is compelling, there are legitimate reasons to question whether it will continue, or that it implies AGI soon.

First, while the tasks tested are much closer to real-world work than most benchmarks, they still need to be well-defined enough to use in a benchmark at all, for example, to have clearly defined success conditions.

To investigate the significance of this drawback, in the full paper METR roughly rated the tasks on how ‘messy’ they were. They found that the messier tasks were indeed harder for AIs (and none of the tasks were as ‘messy’ as something like doing novel research).

However, even among the messier tasks, they observed a similar rate of improvement over time. This suggests AI is still on track to tackle messy tasks, it’s just that it’ll take longer.

Similarly, the horizon was based on when tasks could be completed successfully half the time. If you require a higher chance of completion, the rate of improvement is again similar but lagged by a couple of years.

I expect something similar would be true of high-context tasks: they’re harder to AIs, but context lengths have been steadily expanding over time.

So, we could be in for a future where AI is able to do well-defined one-month tasks with a 50% success by 2030, but still can’t do messier, very high-context ones with higher reliability. Although that could lead to significant automation, human leaders would remain a crucial bottleneck.

Second, the date when AI models reach multi-week tasks is sensitive to the selection of tasks used in the benchmark. METR discussed this objection in their paper, and point out they’re focused on computer use tasks, which they’ve checked across a variety of benchmarks.

METR’s tasks were also been chosen to be especially relevant to automating AI research, which is the class of task that’s most of interest.

But it’s notable AI still can’t reliably do some computer tasks that take humans no time at all; while being able to easily complete tasks that take humans hours (or even decades). So, the notion of a single time horizon is a significant simplification.

Moreover, if we expanded beyond software engineering style computer use tasks, for instance to include robotic manipulation, or the ability to have novel research insights, we might find the trend shows these are still a very long way away.

Update July 2025: METR have subsequently released an expanded data set, finding similar rates of improvement in other domains.

Perhaps the most important objection is that reinforcement learning might work very well for 1-hour tasks, explaining recent progress, but stop working well at some longer horizon.

That’s because for longer horizon, messy, high-context tasks, it’s much harder to create a good reward signal. (It’s also much harder to create a good dataset for pretraining.)

So, maybe at some point in the next few years, this trend will hit a plateau.

In that scenario, we’ll have extremely smart AI assistants, but we won’t be near autonomous AI workers. That would be a pretty good outcome for humanity!

However, if there's one lesson from recent AI progress, it's this: don't bet against straight lines on a graph.

Learn more: METR’s announcement blog post, twitter thread, full paper.

They've missed something huge.

In 2024, while many declared AI was reaching a plateau, it was actually entering a new paradigm: learning to reason using reinforcement learning.

This approach isn’t limited by data, so could deliver beyond-human capabilities in coding and scientific reasoning within two years.

Here's a simple introduction to how it works, and why it's the most important development that most people have missed.

The new paradigm: reinforcement learning

People sometimes say “chatGPT is just next token prediction on the internet”. But that’s never been quite true.

Raw next token prediction produces outputs that are regularly crazy.

GPT only became useful with the addition of what’s called “reinforcement learning from human feedback” (RLHF):

1. The model produces outputs

2. Humans rate those outputs for helpfulness

3. The model is adjusted in a way expected to get a higher rating

A model that’s under RLHF hasn’t been trained only to predict next tokens, it’s been trained to produce whatever output is most helpful to human raters.

Think of the initial large language model (LLM) as containing a foundation of knowledge and concepts. Reinforcement learning is what enables that structure to be turned to a specific end.

Now AI companies are using reinforcement learning in a powerful new way – training models to reason step-by-step:

1. Show the model a problem like a math puzzle.

2. Ask it to produce a chain of reasoning to solve the problem (“chain of thought”).1

3. If the answer is correct, adjust the model to be more like that (“reinforcement”).2

4. Repeat thousands of times.

Before 2023 this didn’t seem to work. If each step of reasoning is too unreliable, then the chains quickly go wrong. Without getting close to correct answers, there was nothing to reinforce.

But now it’s started to work very well…

Reasoning models breakthroughs

Consider GQPA –– a set of new scientific questions designed so that people with PhDs in the field can mostly answer them, but non-experts can’t, even with 30min access to Google. It contains questions like this:

I did a masters level course in theoretical physics, and I have no clue.

In mid 2023, GPT-4 was barley better at random guessing on this benchmark. In other words, it could reason through high school level science problems, but it couldn’t reason through graduate level ones.

Then came GPT-o1, built by OpenAI using reinforcement learning on top of GPT-4o base model.3

Suddenly it could get 70% of questions right – making it about equal to PhDs in the relevant field.

Most people are also not regularly answering PhD-level science questions, so have simply haven’t noticed recent progress.

Most criticisms of AI are based on the free models, and those don’t include o1, which can typically already do the things people say AI can’t do.

And o1 was just the beginning.

A new rate of progress?

At the start of a new paradigm, it’s possible to get gains especially quickly. Just three months later in December, OpenAI released results from GPT-o3 (the second version, but named ‘3’ because o2 is taken by a telecom company).

GPT-o3 is probably GPT-o1 but with even more reinforcement learning, and perhaps the addition of “tree search” – generating 10 or 100 solutions, and picking the one that appears most (yes advancing modern AI really is that simple).4

o3 surpassed human experts on the GPQA benchmark.

(Chart from Ethan Mollick.)

Earlier LLMs were good at writing but bad at math and rigorous thinking. Reinforcement learning flips this pattern – it’s most useful in domains with verifiable answers, like coding, data analysis and science.

GPT-o3 is much better in all of these domains than its base model.

For example, SWE bench verified is a benchmark of real-world software engineering problems from github that typically take under an hour.

• GPT-4 could, when put into an agent architecture, solve about 20%.

• GPT-o3 could solve over 70%.

This means o3 is basically as good as professional software engineers at completing these discrete tasks.

On competition coding problems, o3 would have ranked within the top 200 human competitors in the world.

The progress in mathematics is maybe even more impressive. On high school competition math questions, o3 leapt up another 20% compared to o1 – a huge gain that might have taken a year ordinarily. Most math benchmarks have now been saturated.

In response, Epoch AI created Frontier Math – a benchmark of insanely hard mathematical problems. Field’s Medalist Terrance Tao said the most difficult 25% of questions were “Extremely challenging”, and that you’d typically need an expert in that branch of mathematics to solve them.

Previous models, including GPT-o1, could hardly solve any of these questions.5 OpenAI claimed that GPT-o3 could solve 25%.6

Reasoning models can check their own thinking, so are less likely to hallucinate or make weird mistakes.

AI researcher Francois Challot was a proponent of the common criticism that LLMs are “just sophisticated search” rather than “real reasoning”. He developed the ARC-AGI benchmark, a series of pattern recognition puzzles a bit like an IQ test, which were relatively easy for humans but hard for LLMs. That is, until o3.7

All these results went entirely unreported in the media. In fact, on the same day as the o3 results, the front page of the Wall Street Journal looked like this:

The WSJ article is about GPT-5, but that misses the point. Even without GPT-5, AI can improve rapidly with reinforcement learning alone.

Why this is just the beginning

In January 2025, DeepSeek replicated many of o1’s results. This got a lot more attention because it was Chinese.

But the bigger story is that reinforcement learning works.

A key thing we learned from Deepseek that even basically the simplest version of it works.8 This suggests there’s a huge amount more to try.

(It’s also why Anthropic and Google also have already been able to train models just as good; in fact Google’s Gemini 2.0 Flash is even cheaper and better than DeepSeek, and was released earlier.)

DeepSeek also reveals its entire chain of reasoning to the user. From this, we can see the sophistication and surprisingly human quality of its reasoning: it’ll reflect on its answers, backtrack when wrong, consider multiple hypotheses, have insights and so on.

OpenAI researcher Sabastian Bubeck noted:

No tactic was given to the model. Everything is emergent. Everything is learned through reinforcement learning. This is insane.

We’re also seeing some generalisation. Nathan Labenz claims GPT-o1 is better at legal reasoning, despite not being trained directly on legal problems.

And it will be possible to apply reinforcement learning to other domains, like business strategy or writing tweets, it’s just the reinforcement signals will be noisier, so it will take longer.

How far can this go?

The compute for the reinforcement learning stage of training DeepSeek likely only cost about $1m.

If it keeps working, OpenAI, Anthropic and Google could now spend $1 billion on the same process, a 1000x scale up.9

One reason it’s possible to scale up 1000x is that the models now generate their own data.

This might sound circular, or likely to result in “model collapse”, but it’s not.

You can ask GPT-o1 to solve 100,000 math problems, then take only the correct solutions, and use them to train the next model.

Because the solutions can be formally verified, you’ve generated more examples of genuinely good reasoning.

In fact, this data is much higher quality than internet data, because it contains the whole chain of reasoning, and is known to be correct (not something the text on the internet is famous for).

This creates a potential flywheel:

1. Model solves problems.

2. Use the solutions to train the next model.10

3. The better model can solve even harder problems.

4. That generates more solutions

5. Repeat.

If the models are already able to do PhD-level reasoning, the next stage would be to push into researcher-level reasoning, then perhaps into insights humans haven’t had yet.

Two more accelerants

On top of that, reasoning models unlock several other ways to improve AI.

First, if you ask them to generate longer chains of reasoning for each question, they produce better answers.

That didn’t use to work because mistakes would compound too quickly, but now OpenAI showed that you can have GPT-o1 think 100-times longer than normal, and get linear increases in accuracy on coding problems.

As reasoning models become more reliable, they will be able to think for longer and longer. Just like a human, this lets them solve more difficult problems even without additional intelligence.

This can “pull forward” more advanced capabilities on especially high-value tasks.

Suppose GPT-o7 can answer a question for $1 in 2028. Instead in 2026 you’ll be able to pay GPT-o5 $100,000 to think 100,000 times longer, and generate the same answer.11

That’s too expensive for most users, but still a bargain for important scientific or engineering questions.

Second, reasoning models could make AI agents work a lot better. Agents are systems that can semi-autonomously complete projects over several days, and are now the top priority of the frontier companies.

Reasoning models make agents more capable because:

• They’re better at planning towards goals.

• They can check their work, improving reliability, which is a huge bottleneck.

We’re starting to see signs of how reasoning models, thinking for longer, and agents all mutually support each other.

Humanity’s Last Exam is a collection of 3,000 questions from 100 fields designed to be at the frontier of human knowledge. The full questions are not available on the internet, but include things like:

GPT-4o could answer 3%, and even GPT-o1 could only answer 9%.

In Feb 2025, OpenAI released a research agent, DeepResearch, which can browse through hundreds of web pages and pdfs, do data analysis, and synthesise the results. It scored 27%.12

All this probably explains the even-more-optimistic-than-usual statements from the AI company leaders that started in December.

In November 2024 the OpenAI’s CEO Sam Altman said:

I can see a path where the work we are doing just keeps compounding and the rate of progress we've made over the last three years continues for the next three or six or nine.

Just a month later after the o3 results, that had morphed to:

We are now confident we know how to build AGI as we have traditionally understood it...We are beginning to turn our aim beyond that, to superintelligence in the true sense of the word.

In January 2025, Anthropic’s CEO Dario Amodei told CNBC:

I’m more confident than I’ve ever been that we’re close to powerful capabilities…A country of genius in a data center…that’s what I think we’re quite likely to get in the next 2-3 years

Even Google DeepMind's more conservative CEO Demis Hassabis moved from "maybe 10 years away" to "probably 3-5 years."

They’re probably still overoptimistic (as they’ve been in the past), but reinforcement learning plus agents could be a straight shot to AGI in two years.

Most likely, AGI in the sense of an AI that can do most knowledge work tasks better than most humans13 will take longer due to a long tail of real world bottlenecks in reliability, perception, lack of physical presence etc. (Their deployment will also be slowed by compute constraints, inertia and regulation.)

But definitions aside, our default expectation should be for further dramatic progress in capabilities.

In particular, progress could be even faster than the recent trend for domains especially suited to reinforcement learning, like science, coding and math.

It seems quite likely that within two years we have AIs agents with beyond-human abilities in several-hour coding tasks, and that can answer researcher-level math and science questions.

We may see AI starting to figure out problems that have so far eluded humans.

That would already be a huge deal – enough to accelerate technology and scientific research.

But even more importantly, it might take us to AI that can speed up AI research.

The key thing to watch: AI doing AI research

The domains where reinforcement learning excels are exactly those most useful to advancing AI itself.

AI research is:

• Purely virtual (experiments can be done in code)

• Has measurable outcomes.

• Bottlenecked by software engineering

METR has developed a benchmark of difficult AI research engineering problems – the kind of things that real AI researchers tackle daily, like fine tune a model, or predict the result of an experiment.

When put into a simple agent, GPT-o1 and Claude 3.5 Sonnet are already better than human experts when given 2 hours.

Human experts still overtake over longer timeframes (4+ hours), but AI is getting better at longer and longer horizons.

GPT-4o was better when given only 30 minutes – the leap from that to GPT-o1 being better over two hours was a lot faster than many expected.

And we haven’t even seen the results for o3.

Now consider what might happen the next two years:

• GPT-4o replaced with GPT-5 as the base model

• GPT-5 trained to reason with up to ~1000x more reinforcement learning

• This model put into a better agent scaffolding

A continuation of trend could easily bring us to a model that’s better at human experts at AI engineering over 8h or 16h.

That would be quite close to having mid-level engineering employees on demand.

We don’t know how much that would speed up progress, but a modest speed-up could still bring the next advance sooner.

Historical returns to investment in AI research suggest there’s roughly a 50% chance that starts a positive feedback loop in algorithmic progress.

That would continue until diminishing returns are hit, and could take us from “AI engineering agent” to “full AGI” and onto “superintelligence” within a couple of years. Or at a lower bound, billions of science & coding agents thinking 100x human speed.

Even without a pure software feedback loop, we could still see positive feedback loops in chip design: more AI → more funding for chips → more AI capability → repeat. We could easily enter a world where the number of AI agents increases tenfold yearly.

AI researcher agents could be turned to robotics research, relieving one of the main remaining bottlenecks, and then spread into other forms of R&D.

Eventually we’ll see positive feedback loops at the level of the economy as a whole.

This would be the most important scientific, economic, social and general fate-of-the-world development in the world right now.

I find it extremely surreal how maybe 10,000 technologists on twitter have figured this out, but most of the world continues as if nothing is happening.

Here are some thoughts on what it might mean for your own life. Subscribe for upcoming articles on how to help the world navigate this transition.

He concluded these were “a clear sign that it is time to consider investing in different approaches.”

Two years later, GPT-3 could get most of these right.

Marcus wrote a new list of 15 problems GPT-3 couldn’t solve, concluding “more data makes for a better, more fluent approximation to language; it does not make for trustworthy intelligence.”

A year later, GPT-4 could get most of these right.

Now he’s gone one step further, and criticised limitations that have already been overcome.

Last week Marcus put a series of questions into chatGPT, found mistakes, and concluded AGI is an example of “the madness of crowds”.

However, Marcus used the free version, which only includes GPT-4o. That was released in May 2024, an eternity behind the frontier in AI.

More importantly, it’s not a reasoning model, which is where most of the recent progress has been.

For the huge cost of $20 a month, I have access to GPT-o1 (not the most advanced model OpenAI offers, let alone the best that exists).

I asked GPT-o1 the same questions Marcus did and it didn’t make any of the mistakes he spotted.

First he asked it:

Make a table of every state in the US, including population, area and median household income, sorted in order of median household income.

GPT-4o misses out a bunch of states. GPT-o1 lists all 50 (full transcript).

Then he asked for a column added on population density. This also seemed to work fine.

He then made a list of Canadian provinces and asked for a column listing how many vowels were in each name.

I was running out of patience, so asked the same question about the US states. This also worked:

To be clear, there are probably still some mistakes in the data (just as I’d expect from most human assistants). The point is that the errors Marcus identified aren’t showing up.

He goes on to correctly point out that agents aren’t yet working well. (If they were, things would already be nuts.)

And list some other questions o1 can already handle.

Reasoning models are much better at these kinds of tasks, because they can double check their work.

However, they’re still fundamentally based on LLMs – just with a bunch of extra reinforcement learning.

Marcus’ Twitter bio is “Warned everyone in 2022 that scaling would run out.” I agree scaling will run out at some point, but it clearly hasn’t yet.

Much criticism of AI makes this same mistake – it uses free models that are behind the frontier, and have weaknesses that have already been addressed. Or that get addressed in the next generation.

And rather than looking backward at current limitations, I’m more interested to look forward: what’s the rate of progress and where might this all be heading?

I’ll be writing about that shortly.