Real time access to camera images

[imageProcessing]

Before I purchase a Mavic drone, I wanted to see if it is possible to access camera images in real time, to do image processing tasks like recognize if an object is in the drone's field of view. I don't need the raw data file; low-resolution images are fine. What are the options for doing this? Ideally I could stream the images onto my computer and use software like Matlab do tasks like this example. Note I am not seeking to just view the images, I want to be able to access the actual data (JPEG etc.) in real time. If this is not possible, is it instead possible to have the drone fly to a point, take a still image and send the image file to me, then fly back to the same point and take a second image and send me the image?

 

[Kilrah]

Sounds like you need to develop your own app around the DJI SDK. Obviously that will give you the live feed and everything else.

 

[Meta4]

image processing tasks like recognize if an object is in the drone's field of view.

Can you explain what you mean some other way because that doesn't make much sense.
What kind of object do you want to be able to tell if it's in the drone's field of view that you can't see in the live view before you squeeze the trigger button?

 

[imageProcessing]

@Meta4, Sorry I was not clear. I want to be able to use Matlab's machine vision toolbox on the live image stream from the drone camera to do tasks like automatically identify red objects from a set of objects of different colors, identify which object in the image is a square, etc. But first I need to figure out how to access the live image stream.
@Kilrah, That sounds right. I will look on the Developer site for 'getting started' guides to accessing the image feed through the SDK.

 

[Keule]

But first I need to figure out how to access the live image stream.

It's a scenario well suited for DJI's Manifold (2). It's basically a lightweight computer attached to a Matrice airframe. You then have direct access to the video stream from the camera and you can process it further with SDK as @Kilrah pointed out.

 

[imageProcessing]

@Keule, On the DJI store site, it says Manifold is not available in my region (US). However the Matrice is, so thanks. Is it possible to access the video stream from my laptop so I could process them there, instead of onboard the drone? Or alternatively, to command it to transmit a still image that I trigger manually which I can then process on my laptop. Sorry for the basic questions, I'm just getting started.

 

[Keule]

it says Manifold is not available in my region (US)

That is possible because DJI is now promoting the newer Manifold 2.
If you are looking for a Manifold, you should look at ebay. (i got mine actually new for a very good price)

Is it possible to get the image stream so I can process them on my laptop instead of onboard the drone

Now it gets cumbersome... The way DJI has provided access to that data is done by Mobile SDK. That SDK is meant to be run on an IOS or Android device. You then have to write a routine to feed that video stream somehow (WiFi, HDMI) to a laptop for further processing.

DJI has also released an SDK for Windows 10. It's currently in Beta state. I have no experience in using this one, so you have to look up the possibilities what the Win SDK can do or not.

The real easier way is done by using a Manifold and the OnBoard SDK. (Doing all computations such as color matching in the Manifold computer in real time)

 

[Ronit Mathur]

That is possible because DJI is now promoting the newer Manifold 2.
If you are looking for a Manifold, you should look at ebay. (i got mine actually new for a very good price)


Now it gets cumbersome... The way DJI has provided access to that data is done by Mobile SDK. That SDK is meant to be run on an IOS or Android device. You then have to write a routine to feed that video stream somehow (WiFi, HDMI) to a laptop for further processing.

DJI has also released an SDK for Windows 10. It's currently in Beta state. I have no experience in using this one, so you have to look up the possibilities what the Win SDK can do or not.

The real easier way is done by using a Manifold and the OnBoard SDK. (Doing all computations such as color matching in the Manifold computer in real time)

I am currently working on a project with the DJI Mavic 3E drone that only has the Mobile SDK and Payload SDK (no Onboard SDK). I have image recognition software code already written and want to put it into the drone's computer so that the software can read in the image from the drone's camera, process the image, and output a signal to a DJI speaker attached to the drone (on or off). The two main issues I am facing right now are:

1.) I don't think I can put in my own code into the drone because the DJI Mavic 3E doesn't have the Onboard SDK. I am also not sure if this is possible at all, even with the Onboard SDK.
2.) I don't think I can take the video/image from the drone's camera and use it in the code I put into the drone. In other words, the video footage will only go the drone's hand-held controller and I can't use it in my code.

Would you know if my plan is possible or have any alternate solutions? Any help is appreciated. Thank you.