VPS glitch over water scared me today! (Logs Available)

[Yaros]

The incident abruptly goes back to normal at approx 220sec when the IMU (1) (red graph) quickly adjust itself with 35 degrees ... which then goes back to agreeing with the presumed correct showing compass.

But the drone keeps behaving erratically until 270sec into the flight.
It is extremely strange that while the drone was over water it has behaving erratically, then at 270sec I push the stick forward, it flies over land and the issue stops immediately!

First ... you said you saw the drift & the rapid yaw change, this indicate that the movements really happened as the unused IMU (0), the compass & the gyro claims in the log.

I didn't see the yaw change immediately because for some reason the gimbal decided to stabilize the yaw and keep looking at the same place it did before the yaw change, now the gimbal was pointing diagonally to the left. Then, when I looked at the drone, I saw it was drifting towards the right and rotating slowly.
After that, I tried to stabilize the drone using the sticks, visually looking at it, not the screen. I brought it back a bit to where it was, but the gimbal kept pointing diagonally, and the drone kind of stabilized for a few seconds (4 or 5 secs) and then got even more unstable, it started going left to right pretty fast and abruptly, and the movement got even weirder when I tried to stabilize that with the sticks. I got confused & scared that it might fail, flyaway, or just fall into the water at that point, and decided to push forward and bring it towards land. In the video, during that moment, a propeller is visible in the top left of the screen, showing that the gimbal was not pointing straight. As soon as it was over land (over the rocks & sand, close to the takeoff location) the strange movement and behavior stopped immediately, the gimbal pointed straight again, and the drone behaved as it should.

 

[Yaros]

By the way, just to mention, I have flown the drone 3 times after this problematic flight, and it behaved just fine, however after that I haven't tried to fly over water, I have only flown over land.

Another thing that is unrelated probably but just want to mention, there was a DJI Osmo Pocket close to the drone when it was taking off, but I don't think it could have created magnetic interference.

 

[slup]

What do you mean by IMU(0) and IMU(1)?
As far as I know, the Mavic Air 2 has only one IMU unit, it is not dual IMU like in the first generation Mavic Air.
Is that correct?

Most probably 2 different computations out from the same single hardware ... which would make this a suspected IMU (1) computational error. If so that more easily explains why the IMU (1) abruptly correct itself & align with the compass ... & why you have had problem free flight after this incident flight.

But the drone keeps behaving erratically until 270sec into the flight.
It is extremely strange that while the drone was over water it has behaving erratically, then at 270sec I push the stick forward, it flies over land and the issue stops immediately!

The only thing that looks suspicious after that the yaw error stopped & the IMU & compass started to agree ...

...is this, coming in at 272sec in the DAT event log

 

[Yaros]

In the screenshot you provided there are errors that look like battery errors, is that fine, or is there a risk of it falling from the sky randomly?
I mean, battery errors are no good!

 

[slup]

Neither...
core recovery
smbus read failed
battery error CMD button status error
...looks good.

They only occurred during 0,045sec though ... but can't say if it indicate anything severe.

But this below is on regular basis written in every DAT log from my own Air1 ... it's always been there in every flight for over 3 years ... haven't caused anything so far.

 

[BudWalker]

I'm still going to go with the cause being a confused VPS. IMU:Lat/Longitude and IMU:Yaw are the result of computations that fuse other data. In particular, these IMU data depend on the VPS data. It's not known, at least by me, when the FC will use the VPS data and to what extent. In this case I'm guessing that the FC is relying heavily on the VPS data when the MA2 is close to the surface and that the wave action causes incorrect velocity computations.

@slup
" I'm not comfortable with that the VIO yaw do agree with a possibly failing IMU ..."
I think this means the VIO data is incorrect and is being reflected in the IMU computations.


In the top plot it can be seen that IMU(1):Yaw follows VIO:Yaw until 223 secs when the MA2 gains altitude and moves back over land. At this point the FC switches and starts to not rely on VIO:Yaw.

Although harder to see, the second plot shows that IMU(1):VelN is following VIO:VelN and not GPS:VelN.

 

[BudWalker]

In the screenshot you provided there are errors that look like battery errors, is that fine, or is there a risk of it falling from the sky randomly?
I mean, battery errors are no good!

I try not to rely too much on the messages seen in the eventLog stream. Often too much imagination is required to determine what is meant by these messages. In this particular case these messages might be helping a software engineer debug some code.

 

[BudWalker]

Most probably 2 different computations out from the same single hardware ... which would make this a suspected IMU (1) computational error.

Yes! I've theorized in the past that IMU(0) and IMU(1) are two different computations based on the same sensor data. Usually, I was politely ignored so I quit promoting this idea.

 

[slup]

@slup
" I'm not comfortable with that the VIO yaw do agree with a possibly failing IMU ..."
I think this means the VIO data is incorrect and is being reflected in the IMU computations.

Yeah, it's plausible... a bit disconcerting though, why ignore other data coming from other sensors, a weak strategy in my opinion. This is the first time I see a sketchy VPS reading causing a yaw error.

I saw that the only thing that was made just there when the IMU corrected it self ... was a ascend, but never thought that the VPS was allowed to have this big of a influence.

 

[BudWalker]

Yeah, it's plausible... a bit disconcerting though, why ignore other data coming from other sensors, a weak strategy in my opinion. This is the first time I see a sketchy VPS reading causing a yaw error.

I saw that the only thing that was made just there when the IMU corrected it self ... was a ascend, but never thought that the VPS was allowed to have this big of a influence.

Determining how the fusing algorithm choses it's inputs and their weighting is probably pretty tricky. There will always be the "corner" cases where the wrong choice is made.

To see some other examples take a look at the incidents referenced in this post.

Incidents caused by an incorrect IMU(1):Yaw

 

[slup]

...take a look at the incidents referenced in this post.

Incidents caused by an incorrect IMU(1):Yaw

Now I know why you jumped on the VPS so fast ... but yeah, if you have seen it, you have seen it.

A bit old cases ... have it been any newer that you know of, and did you manage to replicate the error?

 

[Yaros]

Now I thought, could it be the raising humid particles from the sea affecting the IMU?

 

[slup]

Now I thought, could it be the raising humid particles from the sea affecting the IMU?

Oh no ... if electronics get drained enough with water they will stop working ... humidity (water) will not make computations come out with different results.

All in this case begins with that you disregarded this ...

As you flew in these conditions anyway, it unfortunately did't just render the VPS readings unusable ... instead the reading from the infrared sensor were considered to be true by the FC ... even though both the compass, gyro & another computation channel reported something different.

This wrong VPS reading made the IMU (1) yaw calculation come out with a false value ... this yaw value didn't match the drone direction in reality, which meant in the end that a yaw error up to 75 degrees was created. This yaw error made your drone first drift in a straight line, but as the error grew larger & closer to 90 degrees the path changed to a circular one.

So the take away from this ...

Even though this seems to be very unusual (this is the first case I see during 3 years) a sketchy VPS reading can cause real flyaway situations similar those created by powering on the drone in a magnetic distorted area, which makes the IMU initialization go wrong. The difference is that when the VPS reading is causing the yaw error it's possible to manually get out of the flyaway situation by ascending so the VPS can't get any reading anymore ... if the yaw error originate from a magnetic interfered power on spot that isn't possible.

 

[Yaros]

All in this case begins with that you disregarded this

I know, but I never thought that incorrect VPS readings can cause this behavior. I thought that the drone prioritizes the GPS signal over VPS, or at least does have some kind of checks that it does, and if VPS seems fluctuating or providing unreliable data, it would use GPS Only instead.

a sketchy VPS reading can cause real flyaway situations similar those created by powering on the drone in a magnetic distorted area

Didn't know that! I will keep that in mind!

it's possible to manually get out of the flyaway situation by ascending so the VPS can't get any reading anymore

Yes, I ascended & flew forward over land and the issue stopped immediately!

instead the reading from the infrared sensor were considered to be true by the FC ... even though both the compass, gyro & another computation channel reported something different.

How is that possible, why did the drone decide to listen to the VPS instead of the Barometer & GPS?
I flew low over water multiple times already, and what I noticed is: if you keep the drone in movement it doesn't have time to glitch out and confuse the VPS sensors.

Also, does Sport Mode disable VPS? Or is VPS active always?

For example, check the first 20 seconds of this video I made, here I'm flying at a constant speed (40km/h) low over water, but because of the fast speed, the VPS doesn't have time to glitch out by the water, so it's generally safe, however I needed to disable the obstacle avoidance sensors during that clip because the drone would abruptly stop because it would think that the reflections of the sunlight in the water is an obstacle:

 

[slup]

... why did the drone decide to listen to the VPS instead of the Barometer & GPS?

Only DJI's software developers know that ... it's rare that the FC goes with the wrong data & disregard other that's correct, but it happens.

I flew low over water multiple times already, and what I noticed is: if you keep the drone in movement it doesn't have time to glitch out and confuse the VPS sensors.

It's obvious ... due to the speed a reading can't be done or maintained, it's the same as ascending so the VPS can't make a reading.

Also, does Sport Mode disable VPS? Or is VPS active always?

To my knowledge you can't disable the VPS ... but it's easily tested, just switch to Sport & see if you get a VPS height reading or not.

 

[Kilrah]

As far as I remember at some point DJI switched from installing 2 separate magnetometers to installing just one and using the visual systems as 2nd source of heading via optical flow... might have been with the MA1.
I don't believe it's documented whether it uses the bottom camera, the OA cameras, all of them, some of them sometimes etc, but if all cameras see pretty much undiscernable patterns that move on their own this would be easy to upset. Then you get into the situation of having 2 sources of info and no good way of telling which one to believe other than basically a wild guess.

 

[Yaros]

As far as I remember at some point DJI switched from installing 2 separate magnetometers to installing just one and using the visual systems as 2nd source of heading via optical flow... might have been with the MA1.
I don't believe it's documented whether it uses the bottom camera, the OA cameras, all of them, some of them sometimes etc, but if all cameras see pretty much undiscernable patterns that move on their own this would be easy to upset. Then you get into the situation of having 2 sources of info and no good way of telling which one to believe other than basically a wild guess.

That's no good…
I would like DJI to implement a switch in the app to disable VPS completely like in older models.

 

[slup]

As far as I remember at some point DJI switched from installing 2 separate magnetometers to installing just one and using the visual systems as 2nd source of heading via optical flow... might have been with the MA1.

When it comes to redundancy the Mavic Pro series had 2 physical IMU's & 2 compasses ... then from the Mavic Air 1 they kept the 2 IMU's but went with 1 compass & used the visual system as a second "compass" to indicate yaw ... Then from Mavic Air 2 it's only one physical IMU but with 2 different computation channels & one compass together with again the visual system as a second to indicate yaw.

Here below all yaw indicating values from a DAT log from a Mavic Air 1 ...

In this flight the physical IMU (1) was used (dark green), the red dashed is the unused IMU (0). The purple is the physical compass (with some interference) & below that in blue, the yaw coming from the visual system.

All top 4 graphs show pretty much comparable values & follow movements in a good way.

The lower light green graph is the calculated turns from the gyro ... it have another value than the top 4 as the gyro initialize on 0 degrees & is depicted unwrapped (just adding & subtracting turns on a 360 degree axis) ...the other 4 initialize on the cardinal degrees & are depicted on a +/- 180 degree axis... but shape wise it agrees.


The visual sensor doesn't only keep track on yaw as a redundancy compass ... it also follows pitch & roll. The light green & black dashed graphs comes from the vision system ... note the slight delay when bigger movements are made, the red & blue coming from the IMU is registered a bit earlier.

 

[slup]

...I don't believe it's documented whether it uses the bottom camera, the OA cameras, all of them, some of them...

Easily confirmed though ... for a Mavic Air 1 all comes from the bottom sensor... only.

Here below a Sport mode moment (The same if OA is turned off through the app...) & all visual system telemetry (dashed graphs) is working even though the OA is turned off.

 

[Yaros]

When it comes to redundancy the Mavic Pro series had 2 physical IMU's & 2 compasses ... then from the Mavic Air 1 they kept the 2 IMU's but went with 1 compass & used the visual system as a second "compass" to indicate yaw ... Then from Mavic Air 2 it's only one physical IMU but with 2 different computation channels & one compass together with again the visual system as a second to indicate yaw.

This is what I hate about DJI, they downgrade stuff in newer models of their drones. This is a perfect example of that, waypoints is another example, the app (Good DJI Go vs kinda crappy new DJI Fly), and there are many other factors that contribute to this fact.

I don't mind paying a bit more for redundancy, but removing a compass & IMU is not a good thing to do in newer models.
I wonder how is the situation with the Mavic 3, does it use the same system as Mavic Air 2, with that price?