Mind the AI Gap: Understanding vs. Knowing

Chinese room argument^2,3

Strong AI needs to be able to understand, not just know

Understanding requires reasoning

• Identifying causality is central to our understanding:
  • “We understand the sum total of phenomena, in so far as they, by virtue of an internal principle of causality, are connected with each other throughout.” I. Kant⁶

Counterfactual reasoning

is a hallmark of human thought, enabling the capacity to shift from perceiving the immediate environment to an alternative, imagined perspective. Mental representations of counterfactual possibilities (e.g., imagined past events or future outcomes not yet at hand) provide the basis for learning from past experience, enable planning and prediction, support creativity and insight, and give rise to emotions and social attributions (e.g., regret and blame).⁷

Meanwhile, in the LLM world…

Meanwhile, in the LLM world…

Opening questions revisited

1. Can LLMs help us fill our knowledge gaps? Absolutely.¹⁰
2. Do LLMs think and understand? No evidence for it yet.
   • What does it mean to understand? To infer causality.
   • What does it take to understand? To be able to reason.
3. How can LLMs help us understand topics and acquire new skills, and move beyond mere knowledge and execution?
4. How can we avoid the illusion that LLMs understand, and prevent ourselves from offloading our cognitive effort?