Automatic detection of sheep from M2ED

[guttormsen]

Hello,
I am a computer science student writing a thesis on automatic detection of sheep using M2ED by combining the thermal sensor with the visible camera. I have unfortunately hit a wall in progress and I am not sure if it is due to me misunderstanding the technical specs of the drone or something else.

What I want to ask is if someone knows the VFOV of the thermal sensor and the visible camera?

Best regards
Guttormsen

 

[Keule]

What I want to ask is if someone knows the VFOV of the thermal sensor and the visible camera?

M2ED Visual camera FOV: approx. 85°, Thermal camera: HFOV: 57°

Source: Mavic 2 Enterprise- Specifications, FAQs, Videos, Tutorials, Manuals - DJI

 

[guttormsen]

M2ED Visual camera FOV: approx. 85°, Thermal camera: HFOV: 57°

Source: Mavic 2 Enterprise- Specifications, FAQs, Videos, Tutorials, Manuals - DJI

Thanks for the reply, do you also know the vertical field of view?

 

[Keule]

do you also know the vertical field of view?

Thermal camera: FOV: 57° x 44°

 

[brett8883]

Hello,
I am a computer science student writing a thesis on automatic detection of sheep using M2ED by combining the thermal sensor with the visible camera. I have unfortunately hit a wall in progress and I am not sure if it is due to me misunderstanding the technical specs of the drone or something else.

What I want to ask is if someone knows the VFOV of the thermal sensor and the visible camera?

Best regards
Guttormsen

That’s a sweet project! I bet it could be done. However just a heads up that the thermal camera isn’t included in the DJI SDK so unless you have some Jedi computer skills (you might) there’s no way to have a third party application that can control the thermal camera. You can apparently control the gimbal but you can’t say switch between optical and thermal within a 3rd party app.

May be totally irrelevant to what you are doing but thought I’d just make you aware

 

[guttormsen]

Thermal camera: FOV: 57° x 44°

Thank you! Then I can rule out the Technical specs as an error

That’s a sweet project! I bet it could be done. However just a heads up that the thermal camera isn’t included in the DJI SDK so unless you have some Jedi computer skills (you might) there’s no way to have a third party application that can control the thermal camera. You can apparently control the gimbal but you can’t say switch between optical and thermal within a 3rd party app.

May be totally irrelevant to what you are doing but thought I’d just make you aware

You dont need to Control the thermal camera, you can download the media files (JPG) from the drone using the Windows SDK if im not mistaken, that way you can get the high quality photos instead of a low quality stream. I will in the start Focus on doing a mapping survey of the ground and do post-Processing on the photos instead of realtime, but it should be possible to do it realtime aswell.

 

[Keule]

However just a heads up that the thermal camera isn’t included in the DJI SDK

Mobile SDK 4.9.1 supports the M2ED and there are calls to control the thermal camera, settings etc.

DJI Mobile SDK Documentation

developer.dji.com

 

[brett8883]

Thank you! Then I can rule out the Technical specs as an error




You dont need to Control the thermal camera, you can download the media files (JPG) from the drone using the Windows SDK if im not mistaken, that way you can get the high quality photos instead of a low quality stream. I will in the start Focus on doing a mapping survey of the ground and do post-Processing on the photos instead of realtime, but it should be possible to do it realtime aswell.

Yea that will work

Thank you! Then I can rule out the Technical specs as an error




You dont need to Control the thermal camera, you can download the media files (JPG) from the drone using the Windows SDK if im not mistaken, that way you can get the high quality photos instead of a low quality stream. I will in the start Focus on doing a mapping survey of the ground and do post-Processing on the photos instead of realtime, but it should be possible to do it realtime aswell.

Yea that should work. I forget about the Windows SDK. Good luck to you!

 

[brett8883]

Mobile SDK 4.9.1 supports the M2ED and there are calls to control the thermal camera, settings etc.

DJI Mobile SDK Documentation

developer.dji.com

I hope you are correct but I still don’t see one that is “change the camera state to thermal” or “change the camera state to optical.”

The only other DJI dual camera is the Zenmuse xt2 but it has onboard SDK which does have the ability to switch to secondary video feed. I could have missed it and hope you are right. If so Litchi needs to get that ability ASAP

 

[Keule]

I have a little experience with both airframes.
For technical flying (mapping, thermal use, SAR) i use GS PRO, Litchi is the wrong tool.

 

[guttormsen]

I hope you are correct but I still don’t see one that is “change the camera state to thermal” or “change the camera state to optical.”

The only other DJI dual camera is the Zenmuse xt2 but it has onboard SDK which does have the ability to switch to secondary video feed. I could have missed it and hope you are right. If so Litchi needs to get that ability ASAP

There is only one videofeed for the M2ED and can be changed With setDisplayMode, atleast for the Android SDK.


I suspect my drone of not taking synchronous images or that one of the cameras is installed at an angle which throws my algorithms off.
Sample images OneDrive

 

[brett8883]

I have a little experience with both airframes.
For technical flying (mapping, thermal use, SAR) i use GS PRO, Litchi is the wrong tool.

View attachment 71838

So can you switch the camera to thermal in GS Pro and back?

 

[brett8883]

There is only one videofeed for the M2ED and can be changed With setDisplayMode, atleast for the Android SDK.


I suspect my drone of not taking synchronous images or that one of the cameras is installed at an angle which throws my algorithms off.
Sample images OneDrive

Did you see this guy? There a vertical offset too

 

[guttormsen]

Did you see this guy? There a vertical offset too

Thats really interesting, Im going to try and find out how that Works. Thanks!

 

[Keule]

So can you switch the camera to thermal in GS Pro and back?

Not with the M2ED. But this is not really necessary because there are always 2 Images (Visual & MSX or Visual & IR) created for each shot. The same goes with the video feed: (1 for visual and another for MSX or IR)

 

[Keule]

There a vertical offset too

That offset is used to align the visual image with the IR.
It could be the case, that both produced images are not perfectly aligned to create the MSX image.
The offset 'shifts' either horizontally or vertically the thermal output.

 

[brett8883]

That offset is used to align the visual image with the IR.
It could be the case, that both produced images are not perfectly aligned to create the MSX image.
The offset 'shifts' either horizontally or vertically the thermal output.

Sweet, I took that to be the issue he was describing.

 

[guttormsen]

One issue I have identified is that the cameras are not synchronized. Both cameras take a picture, but only one at a time, resulting in a small delay between them which creates a problem for large movements.

 

[brett8883]

One issue I have identified is that the cameras are not synchronized. Both cameras take a picture, but only one at a time, resulting in a small delay between them which creates a problem for large movements.

Ok I’m just thinking out loud here because I like your project here and I already know lots of applications for it but in looking at your two pictures above I’m not convinced that they were taken at two markedly different times. Look at the way both of your arms necks and legs are aligned.

What it looks like to me is due to the optical cameras wider lens and aspect ratio you are getting barrel distortion indicative of a wide angle lens
HOWEVER, we know that the camera of the regular Mavic 2 Zoom applies lens correction to the JPEG images automatically to correct for this. On the raw files it does not but leaves this meta data imbedded in the .dng so that it can be read by a post processing tool. Some editors automatically apply it others do not.

So if you are imputing that the optical camera is a 85 degree FOV and adjusting for distortion in you algorithm then this might be an issue since the jpeg is already adjusting the distortion to account for this.

Can you take a raw image and then process it into a jpeg without lens corrections and see if it matches up better?

 

[brett8883]