The Chat Box Is a Training Wheel
The next interface shift may be from message exchange to shared, inspectable activity.
A chat box encodes a particular theory of human–machine collaboration: one participant formulates a request, sends it, waits, reads a response, and decides what to do next. Many human activities do not unfold as a sequence of messages. People point, compare, gesture, pause, rearrange objects, and work while talking about something else. The opportunity is not to eliminate chat, but to stop treating it as the only doorway into intelligence.
From requests to shared activity
Imagine a student examining a circuit diagram. Instead of describing a component in text, the student touches it and asks, “Why is this one heating up?” The assistant can ground its answer in the selected component and its context. The system is responding to a shared situation, not only a sentence. This can reduce communication overhead, but it also makes the system's assumptions about context more important.
A wider interaction vocabulary
Direct manipulation lets people select, move, circle, or annotate objects. Voice and gesture can help when hands are occupied. Ambient assistance can surface information without a new prompt. Agents can take bounded actions across tools. Each mode has trade-offs: voice can be socially awkward, ambient assistance can interrupt, and delegation can obscure who made a decision. The interface should make active context and permission boundaries visible.
Context must remain inspectable
A multimodal interface should show what object, document, time range, or activity the assistant is referring to. A highlight around a selected object may be more useful than a long explanation of hidden model reasoning. Users should be able to correct a selection, pause sensing, review what was shared, and choose a less expressive mode. More sensors do not automatically make an interaction more human.
Where this can fail
The system may misidentify an object, mistake a glance for intent, or interpret a pause as a request for help. Multimodal systems can collect more sensitive context than text-only tools. Accessibility matters: gesture-first designs need keyboard, voice, and screen-reader alternatives.
Questions for research
- In which tasks do people prefer pointing or manipulating over describing?
- Does persistent visual grounding improve trust or create false confidence?
- How should an assistant communicate uncertainty about the selected referent?
- Can people switch between chat, voice, selection, and direct manipulation without losing context?
Design hypothesis
The next major interface shift may be from message exchange to shared, inspectable activity. Chat remains valuable, but becomes one mode among several.
Research starting points
- Amershi et al. (2019), Guidelines for Human-AI Interaction, CHI. https://doi.org/10.1145/3290605.3300233
- Shneiderman (2020), Human-Centered Artificial Intelligence: Reliable, Safe & Trustworthy, International Journal of Human–Computer Interaction.