Memo on the grain of truth problem

Frameworks that try to predict reality can fail when they miss crucial details. Frameworks about AI have each failed in turn. For instance:

• Future Shock, Alvin Toffler, 1970: Higher adaptation ability than anticipated, sometimes manifested as constantly shifting goalposts for what is “AGI”.
• The Singularity is Near, Ray Kurzweil, 2005: We aren’t seeing a discrete singularity, but rather continuous improvement. No nanotechnology.
• Superintelligence, Nick Bostrom, 2014, or MIRI, 2010s: AI isn’t a single-minded, fanatical forceful and elegant agent. So far, AI is much more of a tool (Google maps) than an agent (James Bond).
• Age of Em, Robin Hanson, 2016: AI isn’t first achieved by cloning specific humans digitally.
• Reframing superintelligence, Eric Drexler, 2019: LLMs turn out to be generalist models, not many specialist models trading with each other.
• Situational awareness, Leopold Aschenbrenner, 2024: To be determined. Perhaps: An AI race with China isn’t inevitable, instead the US may shoot its own foot while China stands by.

As a result, it doesn’t make sense to trust frameworks too much. From where I’m standing, it makes sense to bake in the ability to rapidly deal with surprises and to prospect possibilities from the bottom up by looking at events in the world.

Definition of the grain of truth problem

What if the correct hypothesis is not in your hypothesis space? Then you may not be able to identify dangers effectively, and you can’t process information in a straightforwardly Bayesian manner.

In judgmental forecasting

I’ve spent much of my professional life working within the judgmental forecasting and Bayesian traditions of knowledge, as presented by Phil Tetlock of E.T. Jaynes.

In the Tetlock tradition, forecasters aim to not be “hedgehogs”, who know one big thing, but rather “foxes”, who know many things. Aspiring forecasters come to know that a totalizing worldview is bad for accuracy. This lesson seems to have been lost somewhere along the way.

In the Bayesian tradition, one can update on the probability of A after learning B is:

$$ P(A | B) = \frac{P(A) \cdot P(B | A)}{P(B)} $$

however, if P(B) ~ 0, this rule breaks. If you didn’t think of A beforehand, you might be in that situation.

Why the grain of truth problem might be common

• A single individual has much less “computational power” than the sum of other individuals in the world he is trying to predict
• Empirically, surprising things happened reasonably often over the last few years
• The full laws of physics aren’t yet known
• Hard to see that Prigozhin would march on Moscow, and then that he would stop
• Hard to see nuclear weapons coming in 1900
  • Hard to see technological progress before it unfolds
• Hard to see how I will solve a problem long before I solve it

Some possible solutions and their limitations

Bound domain of applicability of models: When Nate Silver was predicting the chance of a Biden win, he was actually predicting the chance of a Biden win, conditional on a Biden/Trump race. He could have made that conditioning more apparent. Limitations: Good epistemic habit, but doesn’t solve the underlying problem.

Various mathy tricks: One can notice when the in-model probability is too low, pessimize over relevant hypothesis, set formalisms to allow considering hypothesis over only parts of the world, solve for a special case, etc. Limitation: Technical, may require well-specificed model, solutions often don’t translate well to judgmental forecasting practice.

Try hard to find a grain of truth. Limitations: Ennumerating many hypothesis exhaustively ex-ante tends to either become very messy or remain too abstract, and can fail if a real world event falls in a category boundary. Slicing the world into mutually exclusive options fails when you have imperfect categories and you get something on a fuzzy boundary¹. Observing reality very closely to generate new hypotheses seems like a much better option, but is also costly.

Accept presence of unknowns, invest in rapid action. Good way to cope with the problem in practice, but also costly. Research communities may also have forgotten how to act in the real world.

Overall there doesn’t seem to be a silver bullet, only mitigations that are costly to different degrees.

Takeaways

• Not having a grain of truth is a problem
• It might be systematic
• I t can sometimes be noticed, and partially mitigated through obsessiveness
• Prepare for surprises, don’t hug frameworks too tightly

The above was adapted from a talk I gave at Newspeak House. Thanks to attendees for discussion of various “Dewey Decimal Systems for reality”, and particularly for pointing me in the direction of metarationality.