A "standardized" lens testing community effort?

[beroe]

hi all,

I am really new to this, I am an amateur in drones but been doing photography for many many years.
Its depressing how little hard data there is on comparing drones in terms of image quality depending on lenses, models, lemons versus up-to-spec, etc etc. Some folks know how to test new drones for that, others (like me) are a bit in the fog. Im tired or looking at utube or reading about subjective opinions.

Lets come up with a "standardized" way to evaluate general models, but also provide a way for people to test things they buy in a reliable way? Technical folks, please dont chastise me for saying this, we dont need a true scientific test (I am actually a scientist), just something that works ok and is deployable to anyone and then we can all get feedback, learn and apply. And technical folks can evaluate objectively and prove if its all wrong.

What I will propose is to take photographs, from fixed distances (say 10, 20, 100 ft ? convert to meters too), under day day light of a standardized lens testing chart. For example this:

https://www.graphics.cornell.edu/~westin/misc/ISO_12233-reschart.pdf

Anyone can print it, I am also offering to mail prints to anyone that cannot.

What I will do these days, is test my Mavic 3 (original, with the Hasselblad camera and with the tele lens), from ground position and in flight, using various ISO and apertures. I will also test some DSLR medium and telephoto lenses (Nikon and Zeiss), from the exact same spot. I will share the raw files. Will also do a color chart testing (that may be more difficult for some folks and variable depending on many factors, but why not try).

I am particularly interested to see how the tests will compare the Mavic3, Mavic3 Pro and Air 3 (so I do have an "agenda"), but why not make this exercise useful for all the drone models out there?

Suggestions? I know there are experts out there that can constructively chime in.

cheers,
Mike

 

[SmilingOgre]

I would just be happy to see manufacturers make unaltered files available for review. I've got a Mavic 3 Classic and while I'm very happy with the images it produces I would have felt better about buying it had I a raw still and D-log video files to review up front. "Stunning images, Amazing, High dynamic range, yak, yak, yak" say nothing to me. Once I get a file into either Photoshop or Davinci Resolve, I can make an informed decision.

 

[beroe]

I would just be happy to see manufacturers make unaltered files available for review. I've got a Mavic 3 Classic and while I'm very happy with the images it produces I would have felt better about buying it had I a raw still and D-log video files to review up front. "Stunning images, Amazing, High dynamic range, yak, yak, yak" say nothing to me. Once I get a file into either Photoshop or Davinci Resolve, I can make an informed decision.

Exactly. But the manufacturers dont do that. And even if they did, they will provide their generic spec data, and from what I read they sell plenty of lemons too. Not everyone knows how to spot one, especially in the absence of references.

 

[SmilingOgre]

Exactly. But the manufacturers dont do that. And even if they did, they will provide their generic spec data, and from what I read they sell plenty of lemons too. Not everyone knows how to spot one, especially in the absence of references.

I don't put a lot of faith in spec data. There are so many ways to make that data look good on paper without qualifying those specs. Some specs are simply misrepresented. As far as marketing reels, I do appreciate what a professional 20 year colorist with unlimited resources can do but again it says nothing about what I can get out of the files. As far as lemons go, that's another bag of worms. I bought a Nikon D800 and it did perform to specifications, except, it always focused a tiny bit to the rear of the sensor. No spec will call that out. That's misalignment in manufacture.

 

[Felix le Chat]

I don't put a lot of faith in spec data. There are so many ways to make that data look good on paper without qualifying those specs. Some specs are simply misrepresented. As far as marketing reels, I do appreciate what a professional 20 year colorist with unlimited resources can do but again it says nothing about what I can get out of the files. As far as lemons go, that's another bag of worms. I bought a Nikon D800 and it did perform to specifications, except, it always focused a tiny bit to the rear of the sensor. No spec will call that out. That's misalignment in manufacture.

Agreed. Most of these drones are assembled by piecework workers, that means speed over accuracy. Then there may be a bad component batch that gets assembled and shipped before the problem is identified, there is no such thing as a standard assembled model with either drone cameras or DSLR cameras. It's like the old Eric Morecambe gag... all the right notes are present, but not necessarily in the right order.
It would be nice if the manufacturer published accurate lens data that could be used to construct an accurate DNG LCP, but it's hardly surprising they're close-mouthed about it. Would you fork out 3 grand for a camera if you had a manufacturers sheet that told you the lens was plastic?