VPS glitch over water scared me today! (Logs Available)

[Yaros]

Hey, today I was making a short flight to take a video of my friends near the beach, I hovered the drone about 5 meters above the water, it was fine for about 20 seconds (from 0:20 to 0:45 in the logs), but then it started behaving erratically (0:45 - 3:45 in the logs). It started drifting, but I didn't notice that because I was looking at the screen, for some reason the gimbal was locked looking at the same spot all the time (I wasn't in active track nor spotlight!) but the drone itself was drifting. I looked at the drone when my friend screamed that the drone started flying away, then shortly after I saw the propeller in view, which during my 9 months of flying the Air 2, never happened to me. Also to mention, the height indicator in the app was also showing incorrect results, sometimes negative values, but I was 5 meters up! The VPS Height in the logs shows the correct value.

Note: The sea was wavy, but no splashes that could reach the drone.
Can someone please explain what happened?
Here are the logs: DJI Flight Log Viewer - PhantomHelp.com

 

[FLDave]

According to your log you were in quick shots mode.

 

[FLDave]

What does you video record of the flight show?

 

[Yaros]

According to your log you were in quick shots mode.

I wasn't in quickshots mode until 7:27 in the flight. I was in P-GPS (Normal Mode) during the glitch.

 

[slup]

This has most probably nothing to do with the VPS sensor at all ... unfortunately a decrypted DAT log would have been needed in order to closer see why the FC disregarded the positional change (drift) that was occurring & reported by the GPS.

Looked into an obvious drift section in your log ... the drone drifted 5,2m along the red GPS path. The drift direction was directly equal to the light wind direction.



As seen in below chart no horizontal stick commands was issued at all (dashed graphs). The IMU velocity in X (Northerly) & Y (Easterly) was zero mph (purple & black graph)... But at the same time the GPS reports a constantly change in lat & lon position (grey & magenta graph).


This deviation between the GPS & IMU can also be depicted like below ... the green & purple graph shows position derived through the IMU velocity data ... and the red & blue is the position from the GPS. At approx 114sec the deviation steps over a threshold in DatCon & a position error is indicated as true in the log (purple background color).



Below makes it very clear that the IMU thought that the drone was stationary ... meanwhile the GPS reported that it was moving.

The green path below is coming from the IMU ... the drone was according to the IMU totally stationary at the yellow dot while the GPS reported that the drone drifted 5,2m along the red path (see first pic in this post).



So it's a mystery why the FC disregarded the GPS data ... the drone didn't do anything attitude wise to stop the drift into the wind direction, both pitch & roll mainly remained constant during the drift. The motion was stopped when you applied some negative aileron commands.

The height figures & the vertical velocity throughout the whole flight was according to the log data consistent & matched your throttle inputs very well.

 

[Yaros]

So it's a mystery why the FC disregarded the GPS data ... the drone didn't do anything attitude wise to stop the drift into the wind direction, both pitch & roll mainly remained constant during the drift.

This can't be because VPS tried to stabilize by looking at the movement of the waves and getting confused?

The motion was stopped when you applied some negative aileron commands.

Yes, when I noticed that the drone was trying to fly away I tried to fight it with stick commands opposite to the drift.
Then, some seconds after that the fluctuations in the drone's movement got extreme, it started tilting fast side to side, then I got scared and applied throttle up and pitch forward to bring it closer to land.

unfortunately a decrypted DAT log would have been needed in order to closer see why the FC disregarded the positional change (drift) that was occurring & reported by the GPS.

There is no way for me to provide that, right?

Got some additional questions:
1. Which software was used for this log inspection?
2. Why does the VPS Altitude in the logs show incorrect values when it is “hovering” over the water?

 

[slup]

This can't be because VPS tried to stabilize by looking at the movement of the waves and getting confused?

If the GPS lock is bad a drift can occur if the VPS sensor sees the ground beneath & the ground is moving ... like flowing water or waves on water ... but your drone had a high quality GPS lock so a drift like that shouldn't be able to happen as the FC will get changing GPS data in the hover & will counter.

There is no way for me to provide that, right?

You can provide me with a mobile DAT log ... but I can't read it as it's encrypted ... only DJI can.

Got some additional questions:
1. Which software was used for this log inspection?
2. Why does the VPS Altitude in the logs show incorrect values when it is “hovering” over the water?

1. CsvView

2. The VPS height can be somewhat unreliable if hovering over a reflective surface, like water, it's all described in the user manual. But in general the height figures in the log you provided behave reasonable well ... nowhere in the log is negative VPS heights recorded as you said. Below a section of the flight from the place where you stopped the drift I went through earlier ...

The vertical z speed corresponds well with your throttle input & is also reflected ok in the barometric height ... the blue VPS height climbs up though, but only by 5ft here ... I put that to the reflective water surface making it hard for the sensor.

 

[Yaros]

Thank you for the amazing analysis and explanation.
This seems strange to me, that with GPS lock it still drifted.
But what seems even more strange to me is that when the drone started drifting on the yaw axis, the gimbal was still looking at the initial position, so the gimbal was pointed diagonally without me noticing. The drone wasn't in spotlight, and I didn't move the gimbal myself by holding the finger on the screen and then moving. The gimbal somehow tried to correct for the drift or the drone. It looks like the drone itself referenced the IMU for movement, but the gimbal was using the GPS data to point the direction of the camera.
When I got scared and pushed forward, the drone started flying forward, but on video it looked like it flew diagonally, because the gimbal was pointed diagonally. Also, when I pushed forward, because the gimbal was not pointed straight, I saw the propeller in view, which also scared me a bit.

I don't have the video on me now, but might upload it later.

nowhere in the log is negative VPS heights recorded as you said

Sorry, I meant IMU altitude in Phantomhelp log viewer. I could have not been in negative altitude there because I would go under the water in that case.
Here you can see what I'm talking about:

 

[slup]

...This seems strange to me, that with GPS lock it still drifted.

If the IMU or FC doesn't work as intended strange thing's happen. If this was a temporary computational error or if the same will cause troubles in the future again is hard to say.

But what seems even more strange to me is that when the drone started drifting on the yaw axis, the gimbal was still looking at the initial position...

The moment you're speaking of starts 222sec into the flight ... there the gimbal yaw remains but the drone yaw change CCW uncommanded.

... I meant IMU altitude in Phantomhelp log viewer. I could have not been in negative altitude there because I would go under the water in that case.
Here you can see what I'm talking about:

View attachment 149060

The barometric height is relative where you took off ... so if the height is -12,5ft, the drone is 12,5ft below the point you took off from.

Looking in Google Earth ... the HP is indicated to be 3m above sea level ... 12,5ft = 3,8m. So that doesn't seem to be especially unrealistic. What seems to be wrong though, is that the VPS height was 9,8ft, as that would make the height between the HP & sea to be something like 22,3ft ... or 6,8m, the height of the cliff you took off from doesn't seem to be that high from the pictures in the log.

 

[mark.pinskey]

This has most probably nothing to do with the VPS sensor at all ... unfortunately a decrypted DAT log would have been needed in order to closer see why the FC disregarded the positional change (drift) that was occurring & reported by the GPS.

Looked into an obvious drift section in your log ... the drone drifted 5,2m along the red GPS path. The drift direction was directly equal to the light wind direction.

View attachment 149048

As seen in below chart no horizontal stick commands was issued at all (dashed graphs). The IMU velocity in X (Northerly) & Y (Easterly) was zero mph (purple & black graph)... But at the same time the GPS reports a constantly change in lat & lon position (grey & magenta graph).

View attachment 149049
This deviation between the GPS & IMU can also be depicted like below ... the green & purple graph shows position derived through the IMU velocity data ... and the red & blue is the position from the GPS. At approx 114sec the deviation steps over a threshold in DatCon & a position error is indicated as true in the log (purple background color).

View attachment 149050

Below makes it very clear that the IMU thought that the drone was stationary ... meanwhile the GPS reported that it was moving.

The green path below is coming from the IMU ... the drone was according to the IMU totally stationary at the yellow dot while the GPS reported that the drone drifted 5,2m along the red path (see first pic in this post).

View attachment 149051

So it's a mystery why the FC disregarded the GPS data ... the drone didn't do anything attitude wise to stop the drift into the wind direction, both pitch & roll mainly remained constant during the drift. The motion was stopped when you applied some negative aileron commands.

The height figures & the vertical velocity throughout the whole flight was according to the log data consistent & matched your throttle inputs very well.

That is an awesome analysis….Tell me sir, how can I become as proficient as you….What do I read and what exactly can I study?

 

[Capt KO]

I would recommend doing an IMU calibration even if not required. Fixes problems.

 

[slup]

...What do I read and what exactly can I study?

Oh ... unfortunately it isn't an easy way (said without any intention to be cocky ).

The precondition is that you have time to invest & enough interest to keep on learning.

Step 1 will be to learn what different logs these DJI gadgets can produce, what they can contain & which can be read by us. For that I recommend this write up here at the forum --> Mavic Flight Log Retrieval and Analysis Guide

Step 2 will be to start following all kinds of incident cases in this forum section --> Mavic Crash & Flyaway Assistance . There you try to replicate the investigations made, with the same tools used, ask questions if conclusions aren't understood or why certain data is chosen & why it points into a certain direction or reveal a root cause for the incident.

During this, certain tools will be used ... the simplest is web based & will be much like the log player you have in the Fly or GO4 app, just showing heights, speeds, stick commands & such together with a sat picture showing the flight path.

The main 2 here is:
DJI Flight Log Viewer | Phantom Help
Drone Data Management and Flight Analysis | Airdata UAV (require registration but is free up to 100 flights)

In order to be able to perform deeper analyzes out from all the data included in the log a tool like CsvView is usually needed, get it here for free --> CsvView Downloads . The .csv file provided from Airdata & PhantomHelp out from the actual log can also be investigated through Excel.

The tricky part with a .csv file in Excel or in CsvView is that nothing is served on a plate there ... all data is available but you need to chose what data to look at that might lead to the root cause for the incident, so certain knowledge is needed to being able to pick the right thing's to look at based on the incident.

Above all this with logs & tools to read them, knowledge about how & why these drones fly are needed ... what strategies have DJI used when compiling the firmware's... & a lot more.

So yeah ... it will take some time, but gradually you will learn.

Good luck

 

[BudWalker]

This has most probably nothing to do with the VPS sensor at all ... unfortunately a decrypted DAT log would have been needed in order to closer see why the FC disregarded the positional change (drift) that was occurring & reported by the GPS.

Looked into an obvious drift section in your log ... the drone drifted 5,2m along the red GPS path. The drift direction was directly equal to the light wind direction.

View attachment 149048

As seen in below chart no horizontal stick commands was issued at all (dashed graphs). The IMU velocity in X (Northerly) & Y (Easterly) was zero mph (purple & black graph)... But at the same time the GPS reports a constantly change in lat & lon position (grey & magenta graph).

View attachment 149049
This deviation between the GPS & IMU can also be depicted like below ... the green & purple graph shows position derived through the IMU velocity data ... and the red & blue is the position from the GPS. At approx 114sec the deviation steps over a threshold in DatCon & a position error is indicated as true in the log (purple background color).

View attachment 149050

Below makes it very clear that the IMU thought that the drone was stationary ... meanwhile the GPS reported that it was moving.

The green path below is coming from the IMU ... the drone was according to the IMU totally stationary at the yellow dot while the GPS reported that the drone drifted 5,2m along the red path (see first pic in this post).

View attachment 149051

So it's a mystery why the FC disregarded the GPS data ... the drone didn't do anything attitude wise to stop the drift into the wind direction, both pitch & roll mainly remained constant during the drift. The motion was stopped when you applied some negative aileron commands.

The height figures & the vertical velocity throughout the whole flight was according to the log data consistent & matched your throttle inputs very well.

Were it not for @Yaros 's description I would have thought that the MA2 was stationary and the FC erroneously had the MA2 drifting. Could be wrong but I believe that OSD:xSpeed and OSD:ySpeed are the GPS receiver velN and velE. Those, in turn, are computed by the GPS receiver from the doppler frequency shifts. These data are almost always correct.

I think OSD:Latitude and OSD:Latitude are probably the result of the fusion algorithm used by the FC. In this case I suspect the vision system had determined that the MA2 was moving NW but it just the waves moving SW. The FC was probably using the vision system results when computing the Latitude/Longitude.

@Yaros is it possible that in this interval the MA2 was not drifting? The MA2 would have not been moving laterally, rather moving away. After this interval the MA2 was moving erratically which could look like it was drifting.

And, the Mavic Air 2 .DAT recorded on the mobile Fly App is not encrypted. If @Yaros could retrieve that we may be able to learn more.

 

[slup]

Aha ... could I have been wrong regarding the DAT Never save logs from earlier incidents so can't check, must rely on my memory ... & this time that might have glitched

 

[Yaros]

If @Yaros could retrieve that we may be able to learn more.

I sync my logs with DJI Servers, they are deleted then, right?

 

[Yaros]

Where do I find the DAT Files?

 

[Yaros]

If @Yaros could retrieve that we may be able to learn more.

Okay, I found the DAT File!
Luckily it didn't get synced with the servers yet, so it was still on my phone.
Here it is!

 

[slup]

Okay, I found the DAT File!...

Don't have time right now to look thoroughly ... just took a quick glance. Will see if I have more time tomorrow ...

The IMU 1 is used during the flight ... and at 220sec into the flight the disagreement between the IMUyaw & the magYaw is over 70 degrees ... enough to create a yaw error.

The blue graph shows the calculated deviation between ...
The IMUyaw, red graph & the magYaw (compass), green graph.

(Click on the picture to make it larger)

 

[slup]

This is a bit of a mystery still ... my belief is that the problems you had originate from a yaw error, meaning the craft points in another direction in reality than the IMU think's. Depending on how big the deviation is, both drifts, straight or circular uncommanded flight path's & high speed flyaways can occur. All these fly patterns comes from that the FC command the wrong motors to hold position but fails doing so, & tries again and again in vain ...

The first sign of that something is happening out from the data ... is that the used IMU (1) (red graph) & the compass yaw (green graph) slowly starts to deviate. Both agree reasonably well up to the point when you leave the cliff & starts to come out over the sea, the deviation (light blue graph) is there approx. 15 degrees which isn't uncommon, this is about 13sec into the flight.

This state continues until approx 44sec ... there the compass yaw (green graph) starts to record an increasing CCW rotation, but the used IMU (1) (red graph) doesn't follow. This slowly increase the yaw angle deviation continuously until approx 120sec ... here your drone have just drifted away straight sideways those 5,2m I mentioned in my earlier post & the yaw deviation have reached 55 degrees.

(Click on the chart to make it larger)


From 120sec to approx 220sec the IMU (1) yaw & compass yaw deviation keeps on increasing & reaches a tad over 75 degrees ... during this period your drone starts to fly uncommanded in a circular path ( shown in the sat pic. below) with an increasing heading speed ... then at 220sec the IMU (1) yaw value suddenly readjust CCW with 35 degrees. Here the deviation is back at a healthy level of approx 13 degrees which stops the uncommanded movements.



So the used IMU (1) & the compass doesn't agree ... which of them are failing & is the recorded movement really happening in reality?

If we put up all about yaw from the log side by side & compare what agrees & what doesn't ...

We see that the unused IMU (0) (blue graph) agrees with the compass (green graph) ... if we also put up the gyro (grey graph) we see that it also agrees with the CCW rotation.

The used IMU (1) (red graph) doesn't agree with those above, but ... it seems to agree with a yaw measurement coming from the VPS sensor called VIO yaw (purple graph).

So jeez ... what make out of this, 3 values favors a CCW rotation & 2 values vote for no yaw rotation?

It's a couple things that make me point the finger to a failing IMU (1) here ...

First ... you said you saw the drift & the rapid yaw change, this indicate that the movements really happened as the unused IMU (0), the compass & the gyro claims in the log.

Secondly ... a failing compass wouldn't make a drone move uncommanded, this as it's the IMU that handles the flight primarily ... a failing compass can influence the IMU by slowly feeding in correction data, but it will take some time & furthermore, the gyro will also feed in adjustments that then will contradict those from a failing compass.

And lastly ... if the gyro failed, that wouldn't make the compass to follow in the same wrong rotation direction if a movement doesn't occur.

Could it have been a wrongly initialized IMU (1) due to powering on the drone in a magnetic interfered location?

2 thing's points against that ...

1. The magnetic field (magenta graph) didn't change after the take off & during the problematic section of the flight ... if it had been a magnetic interfered powering on spot the magnetic field had changed after take off when your drone flew away from the area ... & the compass had returned to showing a correct heading ... but the IMU, already initialized had kept it's wrong heading.

2. The incident abruptly goes back to normal at approx 220sec when the IMU (1) (red graph) quickly adjust itself with 35 degrees ... which then goes back to agreeing with the presumed correct showing compass.

(Click on the chart to make it larger)


If looking into the DAT event log this is found when the straight drift with those 5,2m have happened & when the uncommanded circular flight is about to start ...


And this when the incident have stopped ...


@BudWalker ...

Do you have another conclusion other than a IMU (1) failure/glitch? I'm not comfortable with that the VIO yaw do agree with a possibly failing IMU ...

 

[Yaros]

What do you mean by IMU(0) and IMU(1)?
As far as I know, the Mavic Air 2 has only one IMU unit, it is not dual IMU like in the first generation Mavic Air.
Is that correct?