The most interesting startup idea I've seen recently: AI for epistemics

If transformative AI might come soon and you want to help that go well, one strategy you might adopt is building something that will improve as AI gets more capable.

That way if AI accelerates, your ability to help accelerates too.

Here’s an example: organisations that use AI to improve epistemics – our ability to know what’s true — and make better decisions on that basis.

This was the most interesting impact-oriented entrepreneurial idea I came across when I visited the Bay area in February. (Thank you to Carl Shulman who first suggested it.)

Navigating the deployment of AI is going to involve successfully making many crazy hard judgement calls, such as “what’s the probability this system isn’t aligned” and “what might the economic effects of deployment be?”

Some of these judgement calls will need to be made under a lot of time pressure — especially if we’re seeing 100 years of technological progress in under 5.

Being able to make these kinds of decisions a little bit better could therefore be worth a huge amount. And that’s true given almost any future scenario.

Better decision-making can also potentially help with all other cause areas, which is why 80,000 Hours recommends it as a cause area independent from AI.

So the idea is to set up organisations that use AI to improve forecasting and decision-making in ways that can be eventually applied to these kinds of questions.

In the short term, you can apply these systems to conventional problems, potentially in the for-profit sector, like finance. We seem to be just approaching the point where AI systems might be able to help (e.g. a recent paper found GPT-4 was pretty good at forecasting if fine-tuned). Starting here allows you to gain scale, credibility and resources.

A forecasting bot made by the AI company FutureSearch is making profit on the forecasting platform Manifold. The y-axis shows profit. This suggests it’s better even than collective prediction of the existing human forecasters.

But unlike what a purely profit-motivated entrepreneur would do, you can also try to design your tools such that in an AI crunch moment they’re able to help.

For example, you could develop a free-to-use version for political leaders, so that if a huge decision about AI regulation suddenly needs to be made, they’re already using the tool for other questions.

There are already a handful of projects in this space, but it could eventually be a huge area, so it still seems like very early days.

These projects could have many forms:

• One example of a concrete proposal is using AI to make forecasts, or otherwise better at truthfinding in important domains. On the more qualitative side, we could imagine an AI “decision coach” or consultant that aims to augment human decision-making. Any techniques to make it easier to extract the truth from AI systems could also count, such as relevant kinds of interpretability research and the AI debate or weak-to-strong generalisation approaches to AI alignment.

• I could imagine projects in this area starting in many ways, including a research service within a hedge fund, a research group within an AI company (e.g. focused on optimising systems for truth telling and accuracy), an AI-enabled consultancy (trying to undercut the Big 3), or as a non-profit focused on policy makers.

• Most likely you’d try to fine tune and build scaffolding around existing leading LLMs, though there are also proposals to build LLMs from the bottom-up for forecasting. For example, you could create an LLM that only has data up to 2023, and then train it to predict what happens in 2024.

• There’s a trade-off to be managed between maintaining independence and trustworthiness, vs. having access to leading models and decision-makers in AI companies and making money.

• Some ideas could advance frontier capabilities, so you’d want to think carefully about either how to avoid that, stick to ideas that differentially boost more safety-enhancing aspects of the technology, or be confident any contribution to general capabilities is outweighed by other benefits (when this is justified is a controversial topic with a lot of disagreement). To be a bit more concrete, finding ways to tell when existing frontier models are telling the truth seems less risky than developing new kinds of frontier models that are optimised for forecasting.

• You’ll need to try to develop an approach that won’t be made obsolete by the next generation of leading models, but can instead benefit from further progress in the cutting edge.

I don’t have a fleshed out proposal, this post is more an invitation to explore the space.

The ideal founding team would ideally cover the bases of: (i) forecasting / decision-making expertise (ii) AI expertise (iii) product and entrepreneurial skills (iv) knowledge of an initial user-type, though bear in mind if you have a gap now, you could probably fill it within a year.

If you already see an angle on this idea, it could be best just to try it on a small scale, and then iterate from there.

If not, then my normal advice would be to get started by joining an existing project in the same or an adjacent area (e.g. a forecasting organisation, an AI applications company) that will expose you to ideas and people with relevant skills. Then keep your eyes out for a more concrete problem you could solve. The best startup ideas usually emerge more organically over time in promising areas.

Existing projects:

• Future Search

• Philip Tetlock’s Lab at UPenn

• Danny Halawi at UC Berkeley

• Ought

• Elicit

• (Let me know about more!)

Learn more:

• Mantic Monday 3/11/24 and 2/19/24 by Astral Codex Ten, which summarise some recent developments in this space.

• Project founder, by me on 80,000 Hours, with some general advice on entrepreneurial careers.

If interested, I’d suggest talking to the 80,000 Hours team (who can introduce you to others working on it).