What if artificial general intelligence (AGI) arrives in just a couple of years, triggering an explosion in science and technology that transforms life as we know it?
I think the arguments for looking after your health could be a bit better, since a tech explosion could bring life extension tech much earlier. It's probably easier to prevent damage than to reverse it, so you might get "stuck" in the condition you were when the acceleration started.
Likewise you could argue it's more important to not die in an accident, since you're more likely to be giving up a 1,000 year lifespan than in business-as-usual. Signing up to cryo could also make more sense because we might be much closer to getting it to work.
I feel like this is a truism, or an improper update? Like if we knew it was 20 years until AGI and 30 years until ASI, people would argue: You should prioritize health in order to make sure you make it to ASI.
Short timelines seem to me that they indicate you should mostly ignore health problems that would only affect you in 20+ years, and should prioritize health interventions that keep you alive until then. This is pretty different from generally being healthy, it's a much narrower target.
I suppose some of my thinking here is that I expect uploading will be a thing and will basically make health problems irrelevant, such that "It's probably easier to prevent damage than to reverse it" doesn't apply because uploading just reverses problems.
I'm mostly in the frame of looking for actions that are good in short or long timelines or no AGI. I agree the more confident you are in short timelines, the more to focus on short-term survival / productivity, and the more you could neglect long-term health.
(e.g. work out 20min a day because it gives you more energy, but not 1h per day, which is probably better for long-term health)
Sure, that’s a fine frame, but it’s distinct from wording this as an update following from short timelines. And worth also remembering that “the optimal amount of health investment is not infinite.” Whether one should be doing more or less health investment depends on what they’re currently doing.
In the post, I maybe put too much emphasis on financial capital. It's probably more important to know the right people, or even have fame, which along with political power could still be useful after the transition. That's harder to act on, but maybe (i) do cool stuff (ii) take jobs that let you meet influential people (iii) follow up on connections you've made.
It's possible I should have put standing up for your political rights on the list. Political advocacy has a tiny chance of having an impact, thought the expected value could still be high: https://80000hours.org/articles/is-voting-important/ Though mostly it's about the social impact rather than personal benefit.
Thanks for writing this! Much more useful than simply stating the world is going to change radically in the near future. If you don't mind the question (sorry if it's too direct), do you think doing a PhD is a reasonable way of getting, say, US citizenship for someone who is still an undergrad? Is there some better alternative?
I haven't looked into the best ways to get a US citizenship. Grad school sounds like a reasonable option (if you're sure you can't get sponsored for a visa right away), though I'm not sure it's worth doing grad school *just* to get a citizenship. PhDs look less attractive if AI might come soon, so you'd want to think about US PhD vs. your best alternatives (e.g. getting a job right away).
Thanks for writing this! Glad to see that others are thinking and writing about this topic. One thought that I have is that I would suggest prioritizing life goals, like having children, now, while we still have stability and normalcy in the world. Who knows where the world will be in a few years, and if it will be a hospitable situation for life goals that are totally attainable right now
Maybe we could go after the root of the problem which is the profit over all else system most of the world lives under. In the U.S., Move to Amend and Alliance for Just Money have some demands we could all get behind. Internationally there is the International Movement for Monetary Reform.
- Resolves with a date, I transfer to you 10 k today-$.
- Does not resolve, you transfer to me 10 k today-$.
- Resolves ambiguously, nothing happens.
The resolution date of the bet can be moved such that it would be good for you. I think the bet above would be neutral for you in terms of purchasing power if your median date of superintelligent AI as defined by Metaculus was the end of 2028.
Interesting post Ben. One question immediately jumped out at me when reading your first point: how do we identify people who are ahead of the curve in such times of flux? COVID was arguably more predictable because the underlying epidemiology was understood. AGI and the dynamics leading up to it are much less understood, so identifying signal in that noise seems much more difficult.
That's a great question. Maybe at some point I'll create a list of resources.
Just quickly, there are people who have prescient historically who I follow (e.g. Carl Shulman, Gwern, the people who figured out the scaling laws).
There are also people who seem (to me) to have a good grasp of what's happening and be plugged in e.g. Dwarkesh, Leopold, Nathan Labenz (Cognitive Revolution Podcast).
"More controversially, if it’s not a big cost to you, it could make sense to delay having children by 3 years, and have them later when uncertainty is reduced."
Why does uncertainty point towards having children later?
Reasons for later:
* Can decide against having children if it's better to never have been born with AGI.
* You want to focus on working on AI Safety now.
* Might be able to have a healthier child once AGI arrives.
Reasons for sooner:
* Having children younger has health benefits for the child.
* You get to spend 3 more years with your children.
* You prefer having a family for 7 years than not having a family at all.
* You believe that early childhood will be better in a pre-AGI world (e.g. similar to how pre-internet childhood had some benefits).
Most of the 'reasons for sooner' are true independently of AI progress. Which points towards a shift towards later. But the shift towards later of 3 years seems rather large.
How about: Build friendships and community? This all reads like an individual fighting the oncoming apocalypse. Friendships are one of the largest contributors to your happiness, won’t be replaced by AI, and also contribute to your resilience.
If uploading did become a thing how many people or which people would even get chosen to be uploaded and then potentially used as a cyborg. It's unlikely the average human would be given a chance.
Unless resources and space was of no concern. By that I mean that if AGI or humans find a way to sustain life and inhabit multiple planets or a spaceship(s).
From my perspective taking care of health is paramount because not only will it help us to live longer (even forever with AGI and/or ASI help) but to be sharper of mind and emotions to more clearly think and act on our ever-changing environments with AI on the rise.
The timelines are unknown so it's better to be safer than sorry.
An extra idea:
I think the arguments for looking after your health could be a bit better, since a tech explosion could bring life extension tech much earlier. It's probably easier to prevent damage than to reverse it, so you might get "stuck" in the condition you were when the acceleration started.
Likewise you could argue it's more important to not die in an accident, since you're more likely to be giving up a 1,000 year lifespan than in business-as-usual. Signing up to cryo could also make more sense because we might be much closer to getting it to work.
I feel like this is a truism, or an improper update? Like if we knew it was 20 years until AGI and 30 years until ASI, people would argue: You should prioritize health in order to make sure you make it to ASI.
Short timelines seem to me that they indicate you should mostly ignore health problems that would only affect you in 20+ years, and should prioritize health interventions that keep you alive until then. This is pretty different from generally being healthy, it's a much narrower target.
I suppose some of my thinking here is that I expect uploading will be a thing and will basically make health problems irrelevant, such that "It's probably easier to prevent damage than to reverse it" doesn't apply because uploading just reverses problems.
I'm mostly in the frame of looking for actions that are good in short or long timelines or no AGI. I agree the more confident you are in short timelines, the more to focus on short-term survival / productivity, and the more you could neglect long-term health.
(e.g. work out 20min a day because it gives you more energy, but not 1h per day, which is probably better for long-term health)
Sure, that’s a fine frame, but it’s distinct from wording this as an update following from short timelines. And worth also remembering that “the optimal amount of health investment is not infinite.” Whether one should be doing more or less health investment depends on what they’re currently doing.
Another interesting take on this topic: https://thezvi.substack.com/p/ai-practical-advice-for-the-worried
In the post, I maybe put too much emphasis on financial capital. It's probably more important to know the right people, or even have fame, which along with political power could still be useful after the transition. That's harder to act on, but maybe (i) do cool stuff (ii) take jobs that let you meet influential people (iii) follow up on connections you've made.
It's possible I should have put standing up for your political rights on the list. Political advocacy has a tiny chance of having an impact, thought the expected value could still be high: https://80000hours.org/articles/is-voting-important/ Though mostly it's about the social impact rather than personal benefit.
Thanks for this, cannot stop thinking about it since I read it and am very worried and concerned for the future of humanity.
Thanks for writing this! Much more useful than simply stating the world is going to change radically in the near future. If you don't mind the question (sorry if it's too direct), do you think doing a PhD is a reasonable way of getting, say, US citizenship for someone who is still an undergrad? Is there some better alternative?
I haven't looked into the best ways to get a US citizenship. Grad school sounds like a reasonable option (if you're sure you can't get sponsored for a visa right away), though I'm not sure it's worth doing grad school *just* to get a citizenship. PhDs look less attractive if AI might come soon, so you'd want to think about US PhD vs. your best alternatives (e.g. getting a job right away).
Nate Silver on this topic: https://www.natesilver.net/p/sbsq-17-how-should-you-prepare-for
This post explores why savings might remain relevant post-AGI (also see the upvoted comments for some push back):
https://www.lesswrong.com/posts/KFFaKu27FNugCHFmh/by-default-capital-will-matter-more-than-ever-after-agi
Thanks for writing this! Glad to see that others are thinking and writing about this topic. One thought that I have is that I would suggest prioritizing life goals, like having children, now, while we still have stability and normalcy in the world. Who knows where the world will be in a few years, and if it will be a hospitable situation for life goals that are totally attainable right now
Maybe we could go after the root of the problem which is the profit over all else system most of the world lives under. In the U.S., Move to Amend and Alliance for Just Money have some demands we could all get behind. Internationally there is the International Movement for Monetary Reform.
Thanks for the advice, Ben! Here is a bet I am open to make with people thinking there is a high chance of an intelligence explosion soon. If, until the end of 2028, Metaculus' question about superintelligent AI (https://www.metaculus.com/questions/9062/time-from-weak-agi-to-superintelligence/):
- Resolves with a date, I transfer to you 10 k today-$.
- Does not resolve, you transfer to me 10 k today-$.
- Resolves ambiguously, nothing happens.
The resolution date of the bet can be moved such that it would be good for you. I think the bet above would be neutral for you in terms of purchasing power if your median date of superintelligent AI as defined by Metaculus was the end of 2028.
Interesting post Ben. One question immediately jumped out at me when reading your first point: how do we identify people who are ahead of the curve in such times of flux? COVID was arguably more predictable because the underlying epidemiology was understood. AGI and the dynamics leading up to it are much less understood, so identifying signal in that noise seems much more difficult.
That's a great question. Maybe at some point I'll create a list of resources.
Just quickly, there are people who have prescient historically who I follow (e.g. Carl Shulman, Gwern, the people who figured out the scaling laws).
There are also people who seem (to me) to have a good grasp of what's happening and be plugged in e.g. Dwarkesh, Leopold, Nathan Labenz (Cognitive Revolution Podcast).
Epoch AI is also very useful.
"More controversially, if it’s not a big cost to you, it could make sense to delay having children by 3 years, and have them later when uncertainty is reduced."
Why does uncertainty point towards having children later?
Reasons for later:
* Can decide against having children if it's better to never have been born with AGI.
* You want to focus on working on AI Safety now.
* Might be able to have a healthier child once AGI arrives.
Reasons for sooner:
* Having children younger has health benefits for the child.
* You get to spend 3 more years with your children.
* You prefer having a family for 7 years than not having a family at all.
* You believe that early childhood will be better in a pre-AGI world (e.g. similar to how pre-internet childhood had some benefits).
Most of the 'reasons for sooner' are true independently of AI progress. Which points towards a shift towards later. But the shift towards later of 3 years seems rather large.
How about: Build friendships and community? This all reads like an individual fighting the oncoming apocalypse. Friendships are one of the largest contributors to your happiness, won’t be replaced by AI, and also contribute to your resilience.
I agree with that. I gesture at it here in the article, but maybe should emphasise it more:
“In COVID, it was very helpful to be part of a group, so seek out people you could live with and who are willing to take non-common-sense actions. Investing in your https://80000hours.org/career-guide/how-to-be-successful/#2-if-helpful-make-mental-health-your-top-priority groundedness and ability to deal with stress could also be worth it.”
(The linked article also talks more about it.)
With This
https://gnomebadhi.substack.com/p/upgrade-your-llm?utm_source=share&utm_medium=android&r=77sior
If uploading did become a thing how many people or which people would even get chosen to be uploaded and then potentially used as a cyborg. It's unlikely the average human would be given a chance.
Unless resources and space was of no concern. By that I mean that if AGI or humans find a way to sustain life and inhabit multiple planets or a spaceship(s).
From my perspective taking care of health is paramount because not only will it help us to live longer (even forever with AGI and/or ASI help) but to be sharper of mind and emotions to more clearly think and act on our ever-changing environments with AI on the rise.
The timelines are unknown so it's better to be safer than sorry.
"be prepared or prepare to fail"