Interactive hand gestures for XR agents: what AgentHands shows
Ask a virtual assistant in a mixed-reality workspace to move a panel "over there," and most systems today can only talk back. AgentHands, a research system for AI agents in extended reality (XR), does something different: it points, then places its hand roughly where "there" is. That small physical act is the whole idea behind interactive hand gestures for XR agents, and it solves a problem that speech alone handles poorly, especially when a task is inherently spatial and words like "that one" or "over there" carry all the ambiguity.
In plain terms, AgentHands gives AI-powered XR agents the ability to point, indicate, and gesture toward objects and locations in sync with what they're saying, so a spatial instruction doesn't have to be spelled out entirely in words. The researchers behind it describe the underlying issue as a "mental mapping gap": when an agent has to describe a location or object purely through text or speech, it limits how clearly it can communicate, no matter how articulate the language model behind it is (AgentHands). That framing matters because it reorients the problem. This isn't about making avatars look more lifelike. It's about giving them a second communication channel for information that language struggles to carry efficiently.
AgentHands sits inside a related approach taking shape across XR research: virtual agents that combine speech with gaze, gesture, and vision rather than a voice interface glued onto an animated character (Frontiers in Virtual Reality). The evidence for any of this is still thin, though. The core AgentHands result, published earlier this year, comes from a 12-person study, and a separate 39-person study of social XR avatars from last year found that the type of gesture animation had no measurable effect on how well a conversation went (IEEE TVCG). Small samples and narrow contexts don't invalidate the findings, but they do mean the story here is more "promising early signal" than "solved problem."
Why interactive hand gestures for XR agents need spatial context
Most existing gesture-generation systems build motion from speech alone, without any awareness of what's actually around the agent. In 2024, researchers working on this problem described the result as gestures generated "in a void": the arm movement looks natural, but the system has no way to ground a point or a reference in the surrounding 3D scene (Incorporating Spatial Awareness in Data-Driven Gesture Generation for Virtual Agents). An avatar can wave its hands convincingly while talking about a chart, but it can't actually indicate which chart, or where on the chart, because nothing in its training data ever tied a gesture to a specific location.
That distinction separates two kinds of gestures. Beat gestures are the rhythmic hand movements that accompany the cadence of speech, the kind that make an avatar look alive but carry no real information. Deictic or referential gestures are different: pointing, indicating distance, showing scale, all gestures whose entire purpose is to reference something specific in the environment. The same research identifies this second category as the one existing systems handle poorly, precisely because it requires linking gesture to scene rather than gesture to sound (same source, Incorporating Spatial Awareness).
Building datasets for referential gestures turns out to be its own problem. A research direction introduced last year merges a synthetic pointing-gesture dataset with a VR dialogue dataset called MM-Conv, which captures two-party conversations along with speech, motion, and 3D scene information, and annotates the dialogue with object references (Grounded Gesture Generation: Language, Motion, and Space). Combined, the two sources add up to just over 7.7 hours of synchronized data. That's not much by the standards of modern machine learning, but it's a meaningful step toward training systems that can point at something specific rather than just gesture generically while talking.
This is the gap AgentHands is trying to close: not "can an avatar move its arms naturally," which is largely solved, but "can an agent gesture at the right thing, at the right moment, in a way tied to the actual 3D space it shares with the user."
How AI agents generate context-aware hand gestures
AgentHands' pipeline works by embedding stage directions directly into the language model's output. When the system generates a verbal response, it tags specific words with GestureEvents, each one specifying a gesture type and the parameters needed to perform it, such as what object or location to point at. A runtime parser then converts those tagged events into timed hand poses, driving an animation system that keeps the gesture synchronized with the corresponding word in speech (AgentHands).
Think of it like a script with stage directions written into the dialogue. The language model is the screenwriter, deciding not just what the agent says but when it should point and at what. The animation system is the actor, reading those cues and performing the physical motion on cue. Neither piece does the other's job: the model decides what needs to be communicated nonverbally, and the rendering layer decides how that looks in three-dimensional space.
That pipeline wasn't designed from a blank page. It came out of a taxonomy the researchers distilled from a formative study with 10 participants, meaning the categories of gestures AgentHands can produce were shaped by observing what people actually expected an agent to do, rather than by guessing (AgentHands).
A separate, open-source toolkit called Anthropomorphic AI, described earlier this year, illustrates a related architectural pattern at a larger scale, though the two aren't interchangeable implementations of the same technique. In that system, a language or vision-language model produces body actions, gaze direction, facial expressions, and text together, and a rendering pipeline, including animation, inverse kinematics, and lip-sync, turns those outputs into a moving, speaking avatar (Frontiers in Virtual Reality). The shared pattern across both projects is a division of labor: the language model decides intent, and a separate layer handles execution. That split is likely to show up again and again as this category of tool matures, because it lets developers swap out the animation engine or the model without rebuilding the whole system.
What the evidence says about gesture-aware conversational agents
The direct evidence for AgentHands comes from a within-subjects study of 12 participants, which found that the system increased engagement and made spatially grounded conversations easier to follow compared to a speech-only baseline (AgentHands). Twelve people is a small study, and the researchers frame it as such, but the direction of the result lines up with the theory: when speech alone can't disambiguate which panel someone means or where it should go, a synchronized point does the disambiguating.
A separate, larger study complicates the picture in a useful way. Researchers ran a mixed-design experiment with 39 participants comparing idle avatars, real-time AI-generated co-speech gestures, and motion-captured movement in social XR conversations, and found no statistically significant effect of animation type on conversation quality or on how participants perceived their conversation partner (IEEE TVCG). In that same study, the strongest predictor of a good experience wasn't the gesture style at all. Participants who reported feeling more immersed in the environment also viewed their partner as more human-like and believable, regardless of whether that partner's avatar was gesturing naturally or barely moving. Several participants said they were paying more attention to what was said than to how the avatar moved.
Taken together, these two studies suggest a narrower and more interesting claim than "gestures make XR agents better." Gestures seem to help specifically when they carry information the words can't, as in a spatial task where "that one" needs a physical referent. They don't seem to help much as generic decoration layered on top of a conversation that would go fine without them. That's a meaningfully different conclusion from the industry's usual pitch that more expressive avatars are automatically better, and it's worth taking seriously precisely because it comes from a null result rather than a success story.
The bigger picture: embodied virtual agents and nonverbal communication in XR
AgentHands is one piece of a larger effort to build embodied virtual agents that combine several channels of nonverbal communication at once. Anthropomorphic AI, the toolkit mentioned earlier, supports speech, gaze, gestures, facial expressions, and full-body movement, with swappable components for speech-to-text, text-to-speech, and the underlying language or vision-language model (Frontiers in Virtual Reality). In one demonstration, the agent detects when a user waves, waves back, greets them, and can approach while maintaining a socially appropriate distance, a small but concrete example of the kind of layered social behavior this category of system is aiming for (Frontiers in Virtual Reality).
None of this feels natural if the pieces are slow. In that toolkit's reported setup, speech-to-text processing averaged 5.579 seconds, the underlying Gemini model responded in about 1.06 seconds, and text-to-speech added roughly another 2.3 seconds, a combined delay that's noticeable in real conversation (Frontiers in Virtual Reality). Newer streaming approaches point toward faster turnarounds: a cited benchmark for Gemini's 2.5 Flash Live API reports a median time-to-first-token of 340 to 420 milliseconds, though that figure describes the model's raw response speed in a benchmark test, not an end-to-end measurement of a full XR conversation pipeline (Frontiers in Virtual Reality). The same toolkit's end-user testing also surfaced a more human problem: current speech-to-text models struggled with strong accents, occasionally frustrating users and producing garbled responses downstream (Frontiers in Virtual Reality). Whether similar gaps exist across different languages, cultural gesture norms, or hardware setups is an open design question the current research doesn't directly address.
Data scarcity compounds the latency problem. One widely used co-speech gesture dataset is built from recordings of a single English speaker, and when researchers needed pointing-gesture data, they had only about 140 original clips to work with, which they stretched to 1,160 through mirroring and time manipulation because no larger equivalent dataset existed (Incorporating Spatial Awareness). In 2025, a field-wide systematic review of embodied conversational agents in XR started with 1,717 candidate papers and, after screening, found only 23 that met its criteria, a result that reflects how selective that review's screening process was rather than the total size of the field, though it still points to how much of the surrounding literature sits outside a narrow, well-defined core (IEEE Access).
Interactive hand gestures for XR agents don't exist in isolation. Language generation, animation, scene sensing, and speech recognition all have to work together, in close to real time, before a gesture lands the way it's supposed to.
Where this leaves the technology
Spatially grounded gestures address a real gap that speech-only agents run into whenever a task depends on location or reference, not just description (AgentHands). The strongest evidence for that claim so far comes from a small comparative study, not from proof that gesture animation broadly improves social XR interactions; a larger study found the opposite for generic gesture style, tying better outcomes to immersion instead (IEEE TVCG). The surrounding infrastructure, from datasets to processing latency to speech recognition performance across accents, remains an open question mark on top of the gesture-generation idea itself, not a solved prerequisite (Frontiers in Virtual Reality).
If larger and more diverse grounded-gesture datasets emerge, and if diffusion-based motion models mature the way early proof-of-concept work suggests they might, the interesting question stops being whether agents can gesture at all (Grounded Gesture Generation). It becomes whether they can do it accurately, across languages and accessibility needs, without pointing at the wrong object at the wrong moment. Readers who want to go deeper can follow the AgentHands paper directly, or track the broader literature on embodied conversational agents to see how XR platforms are converging, unevenly and still early, on agents that use their hands as part of the conversation rather than as an afterthought.
Comments
Be the first, drop a comment!