Ask a virtual assistant in a mixed-reality workspace to move a panel "over there," and a speech-only system can only talk back. AgentHands, a research system for AI agents in extended reality (XR), does something different: it points, then places its hand roughly where "there" is.
That small physical act is the whole idea behind interactive hand gestures for XR agents, and it solves a problem that speech alone handles poorly, especially when a task is inherently spatial and words like "that one" or "over there" carry all the ambiguity.
In plain terms, AgentHands gives AI-powered XR agents the ability to point, indicate, and gesture toward objects and locations in sync with what they're saying, so a spatial instruction doesn't have to be spelled out entirely in words. The researchers behind it describe the underlying issue as a "mental mapping gap": when an agent has to describe a location or object purely through text or speech, it limits how clearly it can communicate, no matter how articulate the language model behind it is.
That framing matters because it reorients the problem. This isn't about making avatars look more lifelike. It's about giving them a second communication channel for information that language struggles to carry efficiently.
AgentHands sits inside a related approach taking shape across XR research: virtual agents that combine speech with gaze, gesture, and vision rather than a voice interface glued onto an animated character (Frontiers in Virtual Reality).
The evidence for any of this is still thin, though.
The core AgentHands result, published at CHI 2026, comes from a 12-person study, and a separate 39-person study of social XR avatars from last year found that the type of gesture animation had no statistically significant effect on conversation quality or how participants perceived their conversation partner (IEEE TVCG).
Small samples and narrow contexts don't invalidate the findings, but they do mean the story here is more "promising early signal" than "solved problem."
Why interactive hand gestures for XR agents need spatial context
Many existing gesture-generation systems build motion from speech without awareness of what's actually around the agent. In 2024, researchers working on this problem described the result as gestures generated "in a void": the arm movement can look natural, but the system has no way to ground a point or a reference in the surrounding 3D scene.
An avatar can wave its hands convincingly while talking about a chart, but it can't reliably indicate which chart, or where on the chart, if its gesture-generation system has no representation of that location (Incorporating Spatial Awareness in Data-Driven Gesture Generation for Virtual Agents).
That distinction separates two kinds of gestures. Beat gestures are rhythmic hand movements that accompany the cadence of speech and typically don't point to a specific object or location. Deictic or referential gestures are different: pointing, indicating distance, showing scale, all gestures whose purpose can be to reference something specific in the environment.
The same research identifies this second category as one existing systems handle poorly, precisely because it requires linking gesture to scene rather than gesture to speech alone.
Building datasets for referential gestures turns out to be its own problem. A research direction introduced last year merges a synthetic pointing-gesture dataset with a VR dialogue dataset called MM-Conv, which captures two-party conversations along with speech, motion, and 3D scene information, and annotates the dialogue with object references (Grounded Gesture Generation: Language, Motion, and Space).
Combined, the two sources add up to just over 7.7 hours of synchronized data. That's not much by the standards of modern machine learning, but it's a meaningful step toward training systems that can point at something specific rather than just gesture generically while talking.
This is the gap AgentHands is trying to close: not simply "can an avatar generate plausible arm motion," but "can an agent gesture at the right thing, at the right moment, in a way tied to the actual 3D space it shares with the user."
How AI agents generate context-aware hand gestures
AgentHands' pipeline works by embedding stage directions directly into the language model's output. When the system generates a verbal response, it tags specific words with GestureEvents, each one specifying a gesture type and the parameters needed to perform it, such as what object or location to point at.
A runtime parser then converts those tagged events into timed hand poses and motions, driving an animation system that keeps the gesture synchronized with the corresponding word in speech.
Think of it like a script with stage directions written into the dialogue. The language model is the screenwriter, deciding not just what the agent says but when it should point and at what. The animation system is the actor, reading those cues and performing the physical motion on cue.
Neither piece does the other's job: the model specifies what needs to be communicated nonverbally and the gesture parameters, while the rendering layer turns those instructions into movement in three-dimensional space.
That pipeline wasn't designed from a blank page. It came out of a taxonomy the researchers distilled from a formative study with 10 participants, meaning the categories of gestures AgentHands can produce were shaped by observations from XR and human-computer interaction experts rather than selected arbitrarily.
A separate, open-source toolkit called Anthropomorphic AI, described earlier this year, illustrates a related architectural pattern at a larger scale, though the two aren't interchangeable implementations of the same technique.
In that system, a language or vision-language model produces body actions, gaze direction, facial expressions, and text together, and a rendering pipeline, including animation, inverse kinematics, and lip-sync, turns those outputs into a moving, speaking avatar (Frontiers in Virtual Reality).
The shared pattern across both projects is a division of labor: the language model decides intent and structured behavior, while separate components handle execution. That separation also lets developers change models, speech services, or animation components without rebuilding the entire system.
What the evidence says about gesture-aware conversational agents
The direct evidence for AgentHands comes from a within-subjects study of 12 participants, which found that the system increased engagement and made spatially grounded conversations easier to follow compared to a speech-only baseline.
Participants completed orchid-care and 3D-printer tasks, and the researchers reported significant improvements in locating referenced objects and directions and in understanding actions when the hand gestures were present. Twelve people is a small study, but the direction of the result lines up with the theory: when speech alone can't disambiguate which object someone means or where something should go, a synchronized point can supply that missing spatial reference.
A separate, larger study complicates the picture in a useful way. Researchers ran a mixed-design experiment with 39 participants comparing idle avatars, real-time AI-generated co-speech gestures, and motion-captured movement in social XR conversations, and found no statistically significant effect of animation type on conversation quality or on how participants perceived their conversation partner (IEEE TVCG).
In that same study, the strongest relationship with a good experience wasn't the gesture style at all. Participants who reported feeling more immersed in the environment also viewed their partner as more human-like and believable, regardless of whether that partner's avatar was gesturing naturally or barely moving. Several participants said they were paying more attention to what was said than to how the avatar moved.
Taken together, these two studies suggest a narrower and more interesting claim than "gestures make XR agents better." Gestures appear useful when they carry information the words don't, as in a spatial task where "that one" needs a physical referent.
They don't appear to improve a social conversation simply because the avatar's generic co-speech animation becomes more sophisticated. That's a meaningfully different conclusion from the industry's usual pitch that more expressive avatars are automatically better, and it's worth taking seriously precisely because one of the studies produced a null result for animation type.
The bigger picture: embodied virtual agents and nonverbal communication in XR
AgentHands is one piece of a larger effort to build embodied virtual agents that combine several channels of nonverbal communication at once. Anthropomorphic AI, the toolkit mentioned earlier, supports speech, gaze, gestures, facial expressions, and full-body movement, with swappable components for speech-to-text, text-to-speech, and the underlying language or vision-language model (Frontiers in Virtual Reality).
In one demonstration, the agent detects when a user waves, waves back, greets them, and can approach while maintaining an appropriate social distance, a small but concrete example of the kind of layered social behavior this category of system is aiming for.
None of this feels natural if the pieces are slow. In the toolkit's reported setup, the average speech-to-text interaction and processing time was 5.579 seconds, but the researchers explicitly say they could not measure end-to-end STT latency because transcription was streamed incrementally.
The underlying Gemini model responded in about 1.06 seconds on average, while text-to-speech processing averaged roughly another 2.3 seconds. Those figures therefore should not be added together as a single end-to-end conversation delay.
Streaming approaches point toward faster turnarounds. The paper cites a benchmark for the Gemini 2.5 Flash Live API reporting a median time-to-first-token of 340 to 420 milliseconds, though that figure describes the model's response speed in a benchmark, not an end-to-end measurement of a full XR conversation pipeline.
The same toolkit's end-user testing also surfaced a more human problem: current speech-to-text models struggled with strong accents, occasionally frustrating users and producing incorrect transcriptions that affected downstream responses (Frontiers in Virtual Reality). Whether similar gaps exist across different languages, cultural gesture norms, or hardware setups is an open design question the current research doesn't directly address.
Data scarcity compounds the problem. One co-speech gesture dataset used in the spatial-awareness research is built from recordings of a single English speaker, and when researchers needed pointing-gesture data, they had only about 140 original clips to work with. They expanded those to 1,160 clips through mirroring and time manipulation because no larger equivalent dataset was available in that work (Incorporating Spatial Awareness).
In 2025, a field-wide systematic review of embodied conversational agents in XR started with 1,717 candidate papers and, after screening, found only 23 that met its criteria (IEEE Access). That result reflects the review's strict inclusion criteria rather than the total size of the field, though it also illustrates how much of the surrounding literature sits outside a narrowly defined core of XR systems with embodied, turn-taking conversational agents.
Interactive hand gestures for XR agents don't exist in isolation. Language generation, animation, scene sensing, and speech recognition all have to work together, in close to real time, before a gesture lands the way it's supposed to.
Where this leaves the technology
Spatially grounded gestures address a real gap that speech-only agents run into whenever a task depends on location or reference, not just description. The strongest direct evidence for AgentHands so far comes from a small comparative study, not from proof that gesture animation broadly improves social XR interactions.
A larger study found no significant effect of animation type on social conversation quality or partner perception, while immersion was associated with stronger presence and more human-like perceptions of the partner (IEEE TVCG). The surrounding infrastructure, from datasets to processing latency to speech recognition performance across accents, remains an open question on top of the gesture-generation idea itself, not a solved prerequisite (Frontiers in Virtual Reality).
If larger and more diverse grounded-gesture datasets emerge, and if spatially conditioned motion-generation approaches mature the way early proof-of-concept work suggests they might, the interesting question stops being whether agents can gesture at all (Grounded Gesture Generation).
It becomes whether they can do it accurately, across languages and accessibility needs, without pointing at the wrong object at the wrong moment. Readers who want to go deeper can follow the AgentHands paper directly, or track the broader literature on embodied conversational agents to see how XR platforms are converging, unevenly and still early, on agents that use their hands as part of the conversation rather than as an afterthought.
Comments
Be the first, drop a comment!