Measure is unceasing

There will always be a Voigt-Kampff test

In the film Blade Runner, the Voight-Kampff test is a fictional procedure used to distinguish androids from humans. In the normal course of events, humans and androids are pretty much indistiguishable, except when talking about very specific kinds of emotions and memories.

Similarly, as language models or image-producing neural networks continue to increase in size and rise in capabilities, it seems plausible that there will still be ways of identifying them as such.

Image produced by DALLE-2
Image produced by DALLE-2 with the prompt “Voight-Kampff test”. Note the water mark at the bottom, as well as the inability to produce text.

For example, for image models:

And for language models:

From left to right: original historical image, image of myself, combination of the two produced using DALLE-2 to modify the jacket to also have a white shirt. This is a small-scale example of how the idiosyncrasies that allow us to unmask DALLE-2 do not matter to its ability to produce value: I like the third image a lot and I am using it on my social media profiles.

But much like in the original Blade Runner movie, these details may not really matter for their economic impact, and the fact that a way exists at all of identifying them will be even less relevant. Similarly, the fact that DALLE-2 and other image models have difficulties correctly rendering teeth or objects in relationship to each other doesn’t really reflect their current ability or future potential to replace many thousands of artists, and generally shape the demand curves of art.

I was thinking about this because I was recently forecasting on a question about “AGI”, where “AGI” was defined as a system that: “is capable of passing adversarial Turing test against a top-5% human, who has access to experts.” But such a system might take a really long time to be developed, even if the economic impact of an AI system is pretty great, because such a system might still have its own idiosyncrasies.

Ultimately, this makes me think that nitpicks and gotchas about ways to differentiate humans and machines aren’t just all that relevant to predicting their future impact. What I care about is closer to the real-world impact of these machines.

That’s all for now.