Controllers or hand tracking for driving a robot remotely, which side are you on?

Nov 18, 2026 Last reply: 1 hour ago 6 Replies

First reply

gary.yates

I went the controller route for anything with real weight behind it. My setup is a pair of Touch controllers running an old Meta frame pose stream into a small hobby arm, and the grip button acts as a deadman, so if I flinch or my grip slips the motion stops. With bare hands…

Read the full reply

Been messing around with streaming headset and hand tracking data straight into a robot control setup, and it got me wondering how other people handle the input side. Controllers feel solid and predictable, but gestures feel way more natural when you're trying to mimic what an arm should actualy do. Curious where people land on this. I've also noticed the feed you see in the headset changes how confident you feel. A smooth high framerate view makes you move the robot faster without even thinking about it, while a choppy one makes you second guess every motion. Is that just in my head, or does latency really shape how people teleoperate? Another thing I keep going back and forth on: how much do you trust the tracking data when contact with the real world is on the line? I've seen tracking drop for a split second and it's unsettling even with safeguards in place. Do you folks mostly use this stuff for data collection and safe demos, or are people actually driving hardware around a room with it? Genuinely torn on what the right balance is, so tell me what your setup looks like and why.


I went the controller route for anything with real weight behind it. My setup is a pair of Touch controllers running an old Meta frame pose stream into a small hobby arm, and the grip button acts as a deadman, so if I flinch or my grip slips the motion stops. With bare hands there is no equivalent. I tried hand tracking once for reach poses and lost the skeleton for half a second while my hand was still in the air. The robot kept the last valid pose. Nothing bad happened because I was moving slow, but that moment stuck with me. On the feed question, latency absolutely changes how you move. I measured around 80ms round trip with everything wired, and closer to 150 over wifi. At 80 I work fast and confident, at 150 I creep and hover over the stick. Friends who tried the rig said the same thing without me prompting them. These days the headset is my data collection tool. Anything near the shelf or the cat gets the controllers.

I ended up going a different direction entirely: I never let the headset pose drive the arm directly. My setup converts hand motion into small incremental commands through a rate limiter, so a tracking hiccup just means the arm freezes where it is instead of lunging. You give up some of that natural one-to-one feel, but I stopped flinching at every dropout once the raw pose was decoupled from the output. On the camera feed question, that is absolutely not in your head. I ran the same pick and place route over a wired connection versus streaming over wifi and my time roughly doubled when the view lagged. I was not moving worse, I just paused before every close approach because I did not trust what I was seeing. High framerate reads as proximity confidence even when nothing about the robot changed. One thing that helped more than I expected: a second camera on a monitor showing the arm from behind, because the headset view is blind to anything past the gripper. I use this rig mostly for collecting reach demonstrations rather than live operation, which is honestly where the tracking stuff earns its keep. Recorded data does not care if the skeleton flickered for a frame, so I get the natural motion without the stakes.

I'm mostly lurking here since my setup is way less fancy, just a webcam on a little rover and an old xbox pad, but the latency thing tracks with my expirience too. When my wifi feed stutters I drive slow and hesitant, and when it's clean I zip around without thinking. Never connected the two untill reading this. One thing I'm wondering about with the headset stuff: does stereo depth actually help you judge distances to objects, or do you end up relying on the hand tracking alone? On my flat screen I constantly misjudge how far the rover is from the table edge, seems like 3d view would fix that but I've never tried it.

I don't buy the rate limiter arguement in the second reply. Slowing the commands down just means the arm gets to the wrong place slower. I've had hand tracking report my wrist a few degrees off while the skeleton never dropped, and the arm follows that bad pose faithfully no matter how you chop the motion up. What actually saved me was bounding the workspace and forcing a re-center before any big move, so a bad readout can't go far. One thing I'd want the incremental setup person to answer: when every command is a small step, doesn't a slow drift from tracking wander add up over a minute? It seems like it has to, since each little error still passes through.

This whole thread takes me back to jogs and a wired teach pendant on the old shop robot. You nudged buttons one axis at a time, no gestures, no feed, and it took forever but nothing ever surprised you. What I learned then and still do now: give the robot a sense of its own staleness. If the pose stream quits updating, the arm should hold and then stop, not coast on whatever it last heard. Slow hands recover, fast hands just break stuff quicker. Doesn't matter how fancy the input is.

Update: I ended up moving to controllers whenever the arm is anywhere near contact, and gestures are now only for planning moves and recording demo data. The point about a slightly wrong wrist pose being followed faithfully stuck with me, that was exactly the failure I kept half-seeing. And the webcam rover comment settled the latency question, same confidence drop on a stuttery feed here.

Join the Discussion

Have something to add? Share your thoughts — no account required.

Didn't find your answer?

Ask the community — no account required