Would you buy a 3D FPV VR/AR DJI Drone App for Oculus Quest 1 / 2 / Pro?

[roguestar]

Hi. I'm a drone app developer, and I'm currently targeting an VR FPV implementation on the Quest 1/2/Pro for DJI Mavic 2s + some older models. I'm curious if people in this forum will find a "generalized FPV with VR/AR capabilities" is something that would be useful for you. Also, how much would you buy for a VR FPV app that can utilize all your Mavic 2 + older drones? Here's a list of DJI drones I'm targeting DJI Mobile SDK 4 Supported Aircraft
I'm also thinking of implementing a mission planner + editor, 3D maps, 2D maps, and possibly a "VR cockpit" so you can feel youre piloting your drone in VR. Though some of the DJI cameras only reach 30 FPS, I still think it would be a good FPV experience, especially for Mini or Spark drones. I'm also implementing touch controller interfaces with Quest 3D hand controllers.
Would you all find this kind of VR app, and utilizing a "general FPV headset" that works for more than 1 drone useful? Please let me know!

 

[Kilrah]

Sounds cool, did you get around the connection issues?

 

[roguestar]

Sounds cool, did you get around the connection issues?

Im building the app right now, and haven't gone for long-flight testing quite yet. I'm starting with short range, since I want to make the VR controls perfect (head controls + hand controls, to be specific)

 

[Rusty_SWO]

I would be interested... but very, very cautious. The sensors on the quest are not for use in sunlight...

 

[roguestar]

I would be interested... but very, very cautious. The sensors on the quest are not for use in sunlight...

yeah thats fine. You can just stay in a car and be seated, while controlling a drone. You can also do Sitting Mode on the quest which is the default for having it not worry about sensors.

My focus is getting a VR cockpit environment, while controlling the FPV environment.

 

[Kilrah]

I'd expect what people want most initially is just the video display with basic OSD, the rest could come after.

 

[roguestar]

I'd expect what people want most initially is just the video display with basic OSD, the rest could come after.

This will actually be my "minimal FPV mode". I would still have the Quest Controllers act as an extension of the DJI controllers.

 

[Kilrah]

This will actually be my "minimal FPV mode". I would still have the Quest Controllers act as an extension of the DJI controllers.

Would specifically NOT want that, no way to be precise enough with hands floating in the air, so be sure that's just an option.
Already seen how that works with the DJI Motion Controller, its implementation is pretty good but it's just not appropriate for "serious" use.

 

[roguestar]

Would specifically NOT want that, no way to be precise enough with hands floating in the air, so be sure that's just an option.
Already seen how that works with the DJI Motion Controller, its implementation is pretty good but it's just not appropriate for "serious" use.

I see what you mean. Thanks for your insight on DJI Motion RC.
Im doing the tests with super close range (within 100meters of ground station), and I wanna see how FPV flight can be improved.

If you're curious about my background : www.roguestar.games. Ive been working with drone interface systems for a long time, so this is an area of development that I want to improve.

Also, would you mind elaborating on "appropriate for serious use"? Very much interested in your experience with that.

 

[Kilrah]

Also, would you mind elaborating on "appropriate for serious use"? Very much interested in your experience with that.

For example you want to do an orbit, you hold the sticks in a fixed position with tiny precise adjustments for an extended period of time, easy to do when you've got your hands resting on the controller and it's just the fine finger motor skills at play. VR controllers work great when there's movement but trying to hold a precise position and angle when that involves the entire arm doesn't work great.

I don't mean it'd be unusable, and it'd certainly be fun/entertaining in some ways, there are people who really like the MC, but just that if it was the only option it would be limiting.

Probably a bit biased myself since the only reason I'm flying any of the "camera" drones nowadays is if I have a job to do, it's only the FPV ones for fun and when you're comfortable with sticks in manual/acro mode everything else just feels frustrating and getting in the way instead of improving the experience.

 

[roguestar]

For example you want to do an orbit, you hold the sticks in a fixed position with tiny precise adjustments for an extended period of time, easy to do when you've got your hands resting on the controller and it's just the fine finger motor skills at play. VR controllers work great when there's movement but trying to hold a precise position and angle when that involves the entire arm doesn't work great.

I don't mean it'd be unusable, and it'd certainly be fun/entertaining in some ways, there are people who really like the MC, but just that if it was the only option it would be limiting.

Probably a bit biased myself since the only reason I'm flying any of the "camera" drones nowadays is if I have a job to do, it's only the FPV ones for fun and when you're comfortable with sticks in manual/acro mode everything else just feels frustrating and getting in the way instead of improving the experience.

Yeah, I've seen this too. VR controllers can get exhausting!
Thanks again for your insight. I'm definitely going to make a "Minimal 3D FPV" interface that addresses comfort + stable controls as its core design.

 

[Kilrah]

I think there are control schemes that could be fun, but unfortunately would likely be outside of the level of integration the SDK allows for, would be closer to manual on the FPV/Avata, but those aren't supported at all...

I'd like to try the MC but in something a bit more "unleashed" than what they restrict you to, obviously we don't have access to that though.

 

[roguestar]

>but unfortunately would likely be outside of the level of integration the SDK allows for

This is what I'm currently investigating. You can send a 3D data structure as an event to the virtual controller section of the, and you can poll it around 20hz to 50hz. But im investigating how range affects this.

>I'd like to try the MC but in something a bit more "unleashed" than what they restrict you to, obviously we don't have access to that though.

Most of the drone apps you've encountered are based on a 2D environment of tablets and touchpads. Not a 3D game controller like the Meta Quest.

DJIs Motion RC is just a hardware implementation. The software to toggle into DJI motion RC are offered in the MSDK already.

 

[Kilrah]

DJIs Motion RC is just a hardware implementation. The software to toggle into DJI motion RC are offered in the MSDK already.

That's fine, but do you get direct thrust and yaw/pitch/roll angular velocity control like in manual or is it still restricted to GPS nav? And can you lock gimbal to airframe like in manual? Those are the key things DJI have never shown on their camera drone models.

The frustration from existing implementations is that the many intermediate layers just add too much crud that in the end it doesn't feel right anymore...

 

[roguestar]

but do you get direct thrust and yaw/pitch/roll angular velocity control like in manual or is it still restricted to GPS nav?

Ill be investigating that, thanks for the clue. The MSDK is pretty dense, and I wasn't sure which features are "restricted"

and can you lock gimbal to airframe like in manual? Those are the key things DJI have never shown on their camera drone models.

This is hopefully something to just turn off and on, and it would be pretty silly of DJI if this wasn't in an SDK. It's literally a Boolean. lol

The frustration from existing implementations is that the many intermediate layers just add too much crud that in the end it doesn't feel right anymore...

I agree 100%. I'm developing a fully spec'd "Core 3D FPV" mode, which is the minimal interface. Minimal as in, its all these basic functions that do not go away, and are as natural + comfortable for the FPV VR flight as possible.

 

[Kilrah]

This is hopefully something to just turn off and on, and it would be pretty silly of DJI if this wasn't in an SDK. It's literally a Boolean. lol

Sure, but for example that's never been a feature on any DJI drone until the FPV, so on other / earlier drones either the SDK, gimbal firmware or both may simply not have any implementation for that / not have it enabled.

Also on the camera drones the gimbal is on very flexible shockmounts so it's actually not trivial to lock the angle, it would need to be active if even possible... the hardware matters.

 

[roguestar]

Sure, but for example that's never been a feature on any DJI drone until the FPV, so on other / earlier drones either the SDK, gimbal firmware or both may simply not have any implementation for that / not have it enabled.

Also on the camera drones the gimbal is on very flexible shockmounts so it's actually not trivial to lock the angle, it would need to be active if even possible... the hardware matters.

DJI Mobile SDK 4 Supported Aircraft Luckily I have a list of cameras for MSDK 4.0

most of the drones im targeting have a fixed camera, so it should be easy to see what that camera + gimbal can do. Im pretty sure its possible to lock / unlock them, via software.

 

[Rusty_SWO]

For example you want to do an orbit, you hold the sticks in a fixed position with tiny precise adjustments for an extended period of time, easy to do when you've got your hands resting on the controller and it's just the fine finger motor skills at play. VR controllers work great when there's movement but trying to hold a precise position and angle when that involves the entire arm doesn't work great.

Why would it be harder to use the sticks on the Oculus controllers than the sticks on the DJI controller?

 

[roguestar]

Why would it be harder to use the sticks on the Oculus controllers than the sticks on the DJI controller?

its a matter of control context : the DJI RC doesn't "send signals" outside of the DJI RC, so I can't fetch those events. (or maybe I can?

I have to investigate the SDK for RC events ,if thats even possible.
I don't want the users to be trapped in the DJI RC, while wearing the Quest, which require quest controller input to manipulate.

Ideally you want one set of controls.

VR FPV is a new design space anyway. So im treating the DJI RC + tablet combo as an emergency ground station for the VR FPV app im making, then im using the Quest + quest controllers as its own controller interface.

This is going to be a "2 person operation" for commercial usage. Since US FAA has VLOS rules. And having Quest goggles on, technically break that rule. So you will need a "co pilot" who has the emergency ground station ready.

 

[roguestar]

Why would it be harder to use the sticks on the Oculus controllers than the sticks on the DJI controller?

Also, for the Quest Controller, I think its pretty much the same environment as a DJI controller. Its a matter of making the proper input/output, with sticks.

Im not necessarily going to use all 3D functions for the quest controller. Thats where design and research come in, which is why I am in this forum asking questions