Help - Image recognition inside drone's computer using drone's camera

[Ronit Mathur]

I am currently working on a project with the DJI Mavic 3E drone that only has the Mobile SDK and Payload SDK (no Onboard SDK). I have image recognition software code already written and want to put it into the drone's computer so that the software can read in the image from the drone's camera, process the image, and output a signal to a DJI speaker attached to the drone (on or off) via the Payload SDK. The two main issues I am facing right now are:

1.) I don't think I can put in my own code into the drone because the DJI Mavic 3E doesn't have the Onboard SDK. I am also not sure if this is possible at all, even with the Onboard SDK. How do I put my code into the drone, and if I can, where?

2.) I don't think I can take the video/image from the drone's camera and use it in the code I put into the drone. In other words, the video footage will only go the drone's hand-held controller and I can't use it in my code. Is it possible to read in the footage from the camera into the drone?

Would my plan be possible to do, or are there any alternate solutions? Any help is appreciated. Thank you.

 

[Cafguy]

Thats quite the project! Very Cool, BUT Why use a DJI Drone Seems like alot of unnecessary workarounds when you could just use an open source drone. OR Perhaps Build the unit into a payload module instead of relying on the drones electronics at all. strap it on and BoBs your uncle lol. use a Walksnail VTX or some other transmission besides OS. Walksnail is open as are other Video systems.

 

[Ronit Mathur]

We bought the drone before we realized that we probably couldn't do the plan I mentioned, which was a mistake. We discussed using an external system like you said with a raspberry pi, a pi camera, and some piezo speakers. But, this kind of defeats the purpose of the customizability of the drone which we were looking for. Thank you for the idea; I will definitely look into it. But would you know if I can utilize any of the drone's components for my project?

 

[Felix le Chat]

Thats quite the project! Very Cool, BUT Why use a DJI Drone Seems like alot of unnecessary workarounds when you could just use an open source drone. OR Perhaps Build the unit into a payload module instead of relying on the drones electronics at all. strap it on and BoBs your uncle lol. use a Walksnail VTX or some other transmission besides OS. Walksnail is open as are other Video sytems.

Or, perhaps purchasing one of the older mavics (mav 2 series) that does have an SDK already released? which would still give you a ready released, sophisticated and stable airframe/software platform on which to build... both of which also carry good cameras.

 

[Cafguy]

I cant think of any of the components that would be of any use to you really. Sorry Unless of course you build the box. Its Kinda what I was thinking- A PI in a box ( Or Arduino ) but I would run a VTX or maybe even an old Drones Stack. with some sort of cooling system (maybe a bunch of holes lol) and its own power Maybe a 3s or such. It would look funny like a little box with a big Rat tail lol (antenna)

 

[surfbird]

You cant run any software on the drone.
Why dont you use MSDK? Its made for what you want.

Some drones you can get a root shell, but I don't thin 3E has any yet.

Rooted mavic 1 pro could basically do what you ask, but it involves a lot of work.

 

[anotherlab]

I am currently working on a project with the DJI Mavic 3E drone that only has the Mobile SDK and Payload SDK (no Onboard SDK). I have image recognition software code already written and want to put it into the drone's computer so that the software can read in the image from the drone's camera, process the image, and output a signal to a DJI speaker attached to the drone (on or off) via the Payload SDK. The two main issues I am facing right now are:

1.) I don't think I can put in my own code into the drone because the DJI Mavic 3E doesn't have the Onboard SDK. I am also not sure if this is possible at all, even with the Onboard SDK. How do I put my code into the drone, and if I can, where?

2.) I don't think I can take the video/image from the drone's camera and use it in the code I put into the drone. In other words, the video footage will only go the drone's hand-held controller and I can't use it in my code. Is it possible to read in the footage from the camera into the drone?

Would my plan be possible to do, or are there any alternate solutions? Any help is appreciated. Thank you.

This sounds like you are trying to hammer in a large square peg into a tiny round hole. You might want to consider designing a system around an open-source platform like ArduPilot or PX4. More projects are listed here. Then you can also define a hardware platform that would include an onboard speaker.

There would be a lot more work involved, but you would control what made up the platform and would be limited by the availability of the various parts of the platform.

 

[surfbird]

I am currently working on a project with the DJI Mavic 3E drone that only has the Mobile SDK and Payload SDK (no Onboard SDK). I have image recognition software code already written and want to put it into the drone's computer so that the software can read in the image from the drone's camera, process the image, and output a signal to a DJI speaker attached to the drone (on or off) via the Payload SDK. The two main issues I am facing right now are:

1.) I don't think I can put in my own code into the drone because the DJI Mavic 3E doesn't have the Onboard SDK. I am also not sure if this is possible at all, even with the Onboard SDK. How do I put my code into the drone, and if I can, where?

2.) I don't think I can take the video/image from the drone's camera and use it in the code I put into the drone. In other words, the video footage will only go the drone's hand-held controller and I can't use it in my code. Is it possible to read in the footage from the camera into the drone?

Would my plan be possible to do, or are there any alternate solutions? Any help is appreciated. Thank you.

Alternate solutions would be using the MSDK
It will add 100ms delay, but if you use a high end phone with tensorflow chip it will probably be faster in the end, compared doing it on the drone.