Automatic detection of sheep from M2ED

[guttormsen]

Ok I’m just thinking out loud here because I like your project here and I already know lots of applications for it but in looking at your two pictures above I’m not convinced that they were taken at two markedly different times. Look at the way both of your arms necks and legs are aligned.

What it looks like to me is due to the optical cameras wider lens and aspect ratio you are getting barrel distortion indicative of a wide angle lensView attachment 71850
HOWEVER, we know that the camera of the regular Mavic 2 Zoom applies lens correction to the JPEG images automatically to correct for this. On the raw files it does not but leaves this meta data imbedded in the .dng so that it can be read by a post processing tool. Some editors automatically apply it others do not.

So if you are imputing that the optical camera is a 85 degree FOV and adjusting for distortion in you algorithm then this might be an issue since the jpeg is already adjusting the distortion to account for this.

Can you take a raw image and then process it into a jpeg without lens corrections and see if it matches up better?

View attachment 71855

Good that you like the project and keep thinking out loud!
I did a simple test of waving my arm to confirm that the camera is not 100% syncronized. The pictures below is an image pair where the thermic camera and visual camera sees my hand at two very different places.



However when I accounted for that and tried to analyse pictures where I stood completly still I had better accuracy but was still off. I think you might be right in that barrel distortion has something to do with the misses I experience. Unfortunately the Mavic 2 Enterprise dual can only save JPEG, but I might be able to reverse the lens distortion correction or take it into account for the calculations of positions.

I will have to do some more research on how the cameras and lenses work. Thanks for the input and help so far, I`ll get back to you with the results!

 

[Keule]

I did a simple test of waving my arm to confirm that the camera is not 100% syncronized.

That's because the visual camera captures footage at 30 Hz and the thermal cam captures with 9 Hz.
Depending how fast you are waving the hand, there is a visual difference/delay as seen in your images above.

 

[brett8883]

Good that you like the project and keep thinking out loud!
I did a simple test of waving my arm to confirm that the camera is not 100% syncronized. The pictures below is an image pair where the thermic camera and visual camera sees my hand at two very different places.

View attachment 71909 View attachment 71910

However when I accounted for that and tried to analyse pictures where I stood completly still I had better accuracy but was still off. I think you might be right in that barrel distortion has something to do with the misses I experience. Unfortunately the Mavic 2 Enterprise dual can only save JPEG, but I might be able to reverse the lens distortion correction or take it into account for the calculations of positions.

I will have to do some more research on how the cameras and lenses work. Thanks for the input and help so far, I`ll get back to you with the results!

I did a little forensic analysis for ya and I found a few interesting things.

Exhibit A: The actual size of the thermal image overlaid on the actual size of the optical image

Clearly the size of the optical dwarfs that of the thermal

Exhibit B:
I then did an aspect ratio preserving upscale on the thermal image (so I resized but didn't distort) and aligned the two photos in photoshop by their center pixels. The misalignment isn't that bad at all. Theres a much greater crop factor with the thermal camera I bet that's what you are missing


Exhibit C:
They still are however, unaligned so I placed a red dot in the center of the optical image and a green dot at the center of the thermal image. (edit pretend the dot below is red I uploaded wrong picture but have to go to bed)


Exhibit D: I then dragged the THERMAL image until visually it looked like a match. I never modified the the optical image and I only upscaled the thermal image.(red dot is center of optical green is center of thermal)

Not too shabby eh?

Exhibit E:
Since we know front the documentation that the thermal image can be moved in relation to the optical image on an (40,-40) scale I put down 80 vertical lines and 80 horizontal grid lines. Id say your thermal I'mage needs to come down 4 pegs and to the left 1 peg.(ill let you figure out which way is + and which is -)


Heres a full size uncompressed tiff which is a pixel for pixel match of your optical image with the upscaled thermal image overlaid so you can match up if you do the experiment yourself.

DJI_0555.tiff

drive.google.com

 

[brett8883]

forget about distortion thats not it

 

[RCdancer]

Thank you! Then I can rule out the Technical specs as an error




You dont need to Control the thermal camera, you can download the media files (JPG) from the drone using the Windows SDK if im not mistaken, that way you can get the high quality photos instead of a low quality stream. I will in the start Focus on doing a mapping survey of the ground and do post-Processing on the photos instead of realtime, but it should be possible to do it realtime aswell.

Yes, I believe that's correct. Anytime you shoot video or a picture, both cameras are active. As he says, you can download the media files.

 

[guttormsen]

That's because the visual camera captures footage at 30 Hz and the thermal cam captures with 9 Hz.
Depending how fast you are waving the hand, there is a visual difference/delay as seen in your images above.

I think it is because DJI chooses to take the Visual image first and then take the thermic image which takes much longer time. If they did it the other way around by taking the thermic first and then the Visual the images would appear closer. This is a non-issue for me during this Project since I can stop moving the droen when I take a Picture.

forget about distortion thats not it

No I absolutely think you are on to something on distortions. The offset you created is correct for this image pair, but when ran on multiple Pictures the offset varies depending on where in the Picture boxes are drawn, suggesting it is distortion. I have implemented a simple distortion correction algorithm today for drawing the boxes, and it moved the boxes closer towards the target position. Im still off, but im closer. I have a feeling it could be because I am only distorting the Visual image, not the thermic one (not even sure if it needs distortion?). As long as I apply a padding inside the boxes to make them larger I am getting acceptable results that can be used! I should still do some more Research on which lenses and sensors DJI use to determine how much distortion I should account for, and try it on even larger datasets before concluding, but im happy With the progress it has created!

 

[brett8883]

I think it is because DJI chooses to take the Visual image first and then take the thermic image which takes much longer time. If they did it the other way around by taking the thermic first and then the Visual the images would appear closer. This is a non-issue for me during this Project since I can stop moving the droen when I take a Picture.




No I absolutely think you are on to something on distortions. The offset you created is correct for this image pair, but when ran on multiple Pictures the offset varies depending on where in the Picture boxes are drawn, suggesting it is distortion. I have implemented a simple distortion correction algorithm today for drawing the boxes, and it moved the boxes closer towards the target position. Im still off, but im closer. I have a feeling it could be because I am only distorting the Visual image, not the thermic one (not even sure if it needs distortion?). As long as I apply a padding inside the boxes to make them larger I am getting acceptable results that can be used! I should still do some more Research on which lenses and sensors DJI use to determine how much distortion I should account for, and try it on even larger datasets before concluding, but im happy With the progress it has created!

Cool cool well keep us updated on your progress! It’s a very very interesting and cool idea!

 

[guttormsen]

Cool cool well keep us updated on your progress! It’s a very very interesting and cool idea!

Definetly will!

Since I cannot get ahold of the raw data or information on how DJI has done the distortion correction I am going to calibrate my calculation algorithm. What makes it hard is that it is a combination of thermal and visual imaging and the Objects I use for calibration needs to be clearly visibly in both specters. I did some simple testing With candles, so im going to upscale it to tortches. The idea is to Place alot of tortches at an area and take Pictures of them. That way I can get the positions of them in both the visible and thermal Pictures and add the coordinates to geogebra, creating a polynom that can be used for calibration.

 

[guttormsen]

So I promised I would keep you updated. The torch Experiment failed. Its good data to use for verification, but I could not manage to do calibration With it. I did take some Pictures of a shooting range where they had a leveled and straight cement line (not sure how to describe it). Since it was long and leveled I could for certain say that the thermal Picture is destorted, while the Visual is not. I then found lots of bugs With my undistortion algorithm but managed to fix it. While this is good News I have still not been able to map accurately between the images. I have some ideas on how to move forward that im going to test this weekend.


The original, distorted thermal image



The undistorted image. Ignore the red boxes. The important part is that the bottom line is straight.

 

[brett8883]

So I had a thought on this. Do you really need the high resolution video from the SD card? Why couldn’t you just take screen recording video where the images are already perfectly overlaid and use that for your analysis? It’s gonna be in 1080p which isn’t terrible and is already much better than the thermal resolution to begin with? Seems like it would instantly solve this issue.

 

[guttormsen]

So the project isnt dead but we have pretty much given up on trying to make overlaying work good enough with this camera, it seems its not high enough resolution. The next approach we are testing is to do independent analysis on thermal camera and then on the visual camera, and if we have positive results in both we can assume there actually is a sheep. To do this I have developed a tool where we are able to analyse the thermal pictures, change palettes and read temperature information. I will probably release this soon-ish since DJI ****** us over on not giving us raw thermal data and instead gave us a crappy compressed jpeg without temperature information

So I had a thought on this. Do you really need the high resolution video from the SD card? Why couldn’t you just take screen recording video where the images are already perfectly overlaid and use that for your analysis? It’s gonna be in 1080p which isn’t terrible and is already much better than the thermal resolution to begin with? Seems like it would instantly solve this issue.

That would not work. The sensor on the camera is 160px * 120px, so when they give you output on sd card and realtime on phones they simply scale up the picture by stretching the pixels, so it doesnt actually give us more information, it just makes it easier for the human eyes to see the information.

 

[brett8883]

So the project isnt dead but we have pretty much given up on trying to make overlaying work good enough with this camera, it seems its not high enough resolution. The next approach we are testing is to do independent analysis on thermal camera and then on the visual camera, and if we have positive results in both we can assume there actually is a sheep. To do this I have developed a tool where we are able to analyse the thermal pictures, change palettes and read temperature information. I will probably release this soon-ish since DJI ****** us over on not giving us raw thermal data and instead gave us a crappy compressed jpeg without temperature information

That would not work. The sensor on the camera is 160px * 120px, so when they give you output on sd card and realtime on phones they simply scale up the picture by stretching the pixels, so it doesnt actually give us more information, it just makes it easier for the human eyes to see the information.

Yea but weren’t you doing the same thing to get the thermal to match the size of the RGB photo to overlay? Wouldn’t less but better information be conceivably as good as more lower quality information?

Have you tried it?

 

[bktodd80]

Hello,
I am a computer science student writing a thesis on automatic detection of sheep using M2ED by combining the thermal sensor with the visible camera. I have unfortunately hit a wall in progress and I am not sure if it is due to me misunderstanding the technical specs of the drone or something else.

What I want to ask is if someone knows the VFOV of the thermal sensor and the visible camera?

Best regards
Guttormsen

You should also know that the resolution for the M2ED for the thermal camera is not very robust. I would assume you could combine the visual and thermal picture (like how the Pilot app uses MSX to combine the camera inputs into one image) and use Computer Vision (a field of study within the Artificial Intelligence field) to identify objects in the picture (e.g. sheep).

 

[kcluver]

So the project isnt dead but we have pretty much given up on trying to make overlaying work good enough with this camera, it seems its not high enough resolution. The next approach we are testing is to do independent analysis on thermal camera and then on the visual camera, and if we have positive results in both we can assume there actually is a sheep. To do this I have developed a tool where we are able to analyse the thermal pictures, change palettes and read temperature information. I will probably release this soon-ish since DJI ****** us over on not giving us raw thermal data and instead gave us a crappy compressed jpeg without temperature information





That would not work. The sensor on the camera is 160px * 120px, so when they give you output on sd card and realtime on phones they simply scale up the picture by stretching the pixels, so it doesnt actually give us more information, it just makes it easier for the human eyes to see the information.

Hello guttormsen. May I ask if you managed to develop a tool to analyse the M2ED images? Any chance you would be willing to share?