Point to What You Mean
Reference should be a first-class input, not a puzzle the user has to solve in text.
Human communication is full of references that work because people share a visual or physical context. “That one,” “the second option,” “move it slightly,” and “what is happening here?” are clear when both people can see the same thing. Text-only assistants often make users convert these references into descriptions. That conversion is a hidden interaction cost, especially for diagrams, maps, code, photographs, and complex interfaces.
A shared field of attention
A designer reviewing a screen circles an icon and says, “This feels too prominent.” The circle identifies the referent; the words express an evaluation; the screen supplies context. A useful assistant connects all three. It might propose alternatives, but should not silently change the design. Selection, utterance, and action scope are distinct parts of the interaction.
A deliberate interaction sequence
The person selects an object. The interface visibly confirms it. The person speaks or types an instruction. If ambiguity matters, the system briefly restates its interpretation. It then proposes an edit or performs a low-risk action. This adds slight friction, but protects against high-cost mistakes. Confirmation can be lighter for reversible actions and more explicit for consequential ones.
Not all signals mean the same thing
A click is usually intentional; gaze can be incidental. A gesture may have different meanings across contexts. Touch can be precise but unavailable at a distance. The interface should communicate the source and confidence of a reference. “You selected the left chart” is more useful than silently acting on an uncertain gaze estimate.
Where this can fail
Camera-based interaction may expose bystanders or private documents. Gesture-first interfaces may exclude people with motor differences. Pointing must be optional, configurable, and paired with non-visual alternatives.
Questions for research
- How much time does pointing save compared with verbal description?
- Which selection errors are most common across touch, cursor, gaze, and gesture?
- Do confidence cues help users notice ambiguous references?
- Can the design support screen readers and keyboard-only workflows equally well?
Design hypothesis
A powerful human–AI interface will treat reference as a first-class input and make the target of attention visible before acting.
Research starting points
- Bolt (1980), Put-That-There: Voice and Gesture at the Graphics Interface, SIGGRAPH. https://doi.org/10.1145/800250.807503
- Oviatt (1999), Ten Myths of Multimodal Interaction, Communications of the ACM 42(11).