Low GPS signal / bad vision system data leading to fly-away and crash

[djules75]

This is my first post in this forum.

I initially posted my story on the DJI forum and somebody suggested I post a link here as some people might find this interesting.

The full details are in the DJI post, but long story short:

I lost my drone flying in a famous scenic spot in China (Tiger Leaping Gorge). According to the best analysis from BudWalker: following weak GPS signal, the vision system took over and provided inaccurate speed measurement to the flight controller, leading to a loss of control and my DJI Mini 3 pro crashing into the gorge's cliff wall.

Fly-away / crash - Trying to understand

 

[AeroJ]

Yes this is quite interesting and another timely reminder that flying in low canyons and gorges is rather fraught with additional risk. Sorry to hear you lost one that way. Still, at least you got a fair picture of what probably happened on the DJI forum...

 

[djules75]

Yes, I understand that the risk of low GPS is high in those conditions. However, since it was absolutely not windy, I was expecting to be able to fly it without stabilization at worse, not to completely loose control of it (as I said in the DJI post, I also fly FPV and not having stable positioning doesn't scare me).

But well... that's a costly way to learn...

 

[BudWalker]

I learned a lot with this one. With just the .txt and sketchy GPS data I assumed that it was the GPS doppler velocities that caused the problem. But, then @djules75 provided the .DAT and it was clear that it was the VIO velocities that were incorrect. We’ve seen a lot of incidents were only the .txt is available and either OSD:xSpeed or OSD:ySpeed seemed to be incorrect. It never occurred to me that the cause was incorrect VIO velocities.

Another interesting aspect is that the vision system was off when the Mini3 was flying down river and then turned on when the Mini3 turned and was facing upriver. Pure speculation on my part but I bet that the vision system found it easier to produce velocities, although incorrect, facing against the water flow. Just a theory.

 

[BudWalker]

For anybody wanting to look at this here are the log files

2024-09-17_15-27-13_FLY234.DAT

DJIFlightRecord_2024-09-17_[15-27-59].txt

 

[Zbip57]

@BudWalker Your analysis on the DJI forum was fascinating. Something sure went haywire right after the yaw to point the drone upstream, after which in sticks centred hover it started to drift downstream. It does seem at that point the VPS optical sensor started tracking the moving water.

Despite the then applied full forward right stick, it picked up speed rearward. The subsequent left stick quick input to yaw left./right worked as it should, but the full-left input on the right stick unfortunately came too late.

I made this video with my Mini-1 a year ago, demonstrating how the drone might drift away while in hands-free hover if the VPS optical sensor tracks a moving surface below it, i.e. things like flowing water, or wind ripples on the water's surface, or a floating leaf, etc. I used a patterned rug for this demonstration.

My intent in making the video was to demonstrate that it would ONLY ever behave like this in the ABSENCE of a solid GPS connection. You can see by my reaction in the video that I was STUNNED to discover that's not true at all. Even WITH a solid GPS lock, the Mini insisted on tracking and following the movement as I pulled the rug away!

That seems totally bizarre to me. Surely the IMU inertial measurement unit, plus the GPS monitoring, should sense that the drone is moving despite being commanded to hover in one spot. WHY would DJI prioritize the VPS optical sensor over what the IMU and GPS is saying?

In any case, the issue should only ever arise while in a hands-free hover. The drone should always respond properly to active stick inputs, whether in GPS mode, VPS mode, or ATTI mode. What happened in the Chinese river canyon looks like a serious flaw in the control algorithm.

To view this content we will need your consent to set third party cookies.
For more detailed information, see our cookies page.

Accept third party cookies

 

[BudWalker]

@BudWalker ........

Despite the then applied full forward right stick, it picked up speed rearward. The subsequent left stick quick input to yaw left./right worked as it should, but the full-left input on the right stick unfortunately came too late.
.......

It looks to me like the pitch forward had the proper effect. At 200 secs the Mini4 started to gain NE speed because VIO started tracking the water flow and there were no control inputs. I.e, the Mini 4 thought it was hovering but was actually moving with the water. Full pitch forward wasn't applied until about 208 secs causing the Mini 4 to decrease speed.

 

[djules75]

Similar story here:

To view this content we will need your consent to set third party cookies.
For more detailed information, see our cookies page.

Accept third party cookies

Edit:
Related post: Very Informative Mini 3 Pro Crash Analysis

 

[Zbip57]

Related post: Very Informative Mini 3 Pro Crash Analysis

Thanks for reminding me of that one. I've added a followup post to that thread.

I'm embarrassed to admit how wrong I was in what I posted there 2 years ago. I knew the drone was capable of drifting away while in hover if the VPS optical sensor was tracking a moving pattern. But I was convinced that could ONLY ever happen in the absence of sufficient GPS reception. Turns out, that's not true.

It's certainly more likely to happen in situations of poor GPS reception. But, as shown in my video above, the VPS optical sensor can cause the drift away while in sticks-centred hover if it's tracking floating leaves or bubbles on the surface of moving water etc, EVEN WHEN the drone has a solid GPS lock!

 

[Yorkshire_Pud]

Just a point

I learned a lot with this one. With just the .txt and sketchy GPS data I assumed that it was the GPS doppler velocities that caused the problem. But, then @djules75 provided the .DAT and it was clear that it was the VIO velocities that were incorrect. We’ve seen a lot of incidents were only the .txt is available and either OSD:xSpeed or OSD:ySpeed seemed to be incorrect. It never occurred to me that the cause was incorrect VIO velocities.

Another interesting aspect is that the vision system was off when the Mini3 was flying down river and then turned on when the Mini3 turned and was facing upriver. Pure speculation on my part but I bet that the vision system found it easier to produce velocities, although incorrect, facing against the water flow. Just a theory.

Would you consider this to be a drone fault and if so do you think the OP has a chance in submitting a warranty claim ?

 

[BudWalker]

Just a point

Would you consider this to be a drone fault and if so do you think the OP has a chance in submitting a warranty claim ?

I think this was a deliberate choice made by DJI engineering. So, I don’t see much chance of a successful warranty claim. But, you won’t know if you don’t try.

Which is a greater probability; 1) ground is moving, or 2) location solution isn’t correct. I would choose 2. Aren’t there explicit warnings about flying over water?

And, for this incident, the FC's confidence in the location solution was low.

 

[Yorkshire_Pud]

Aren’t there explicit warnings about flying over water?

There might well be but they might mention "low over water".
l know I am very careful when low over water with a finger ready to give it full throttle if I think something is going wrong but I 've never flown over an enclosed river, I bought the M2Z for a possible flight in such a place in Scotland.

 

[Zbip57]

There might well be but they might mention "low over water".
l know I am very careful when low over water with a finger ready to give it full throttle if I think something is going wrong but I 've never flown over an enclosed river, I bought the M2Z for a possible flight in such a place in Scotland.

There are two different potential issues with "low over water". Here we're only talking about horizontal position hold due to the optical sensor fixating on moving patterns, and nothing to do with vertical hold.

I experienced it once when flying very "low over water" in a deep river gorge. It's in the very opening shot of this video below. I was flying my Mini very low over the river's surface, manoeuvring to get it right into the middle of the gap where the current funnels between the rocks. I paused to hover for a moment to rotate the camera angle the way I wanted, and then was surprised to see the drone moving all by itself following the current flow! Luckily it stopped where the water drops into a stationary foamy pool.

At the time I attributed it to VPS-mode taking over due to poor GPS reception between the steep rock walls. I just never suspected the same thing can also happen even WITH good GPS reception.

The next couple of clips were similarly low over the moving water, with streams of white foam floating past directly below the drone. The gimballed video lens was slowly rotating upwards, but the optical sensor looking straight down is still seeing the exact same stuff moving past directly beneath the drone. Here the drone totally ignores that stuff while I'm driving the drone with active stick inputs.

The issue only arises when the sticks are centred in hover as then the autopilot takes over trying to hold a steady horizontal position.

"Over water" is one thing. Like in my video where I'm dragging the rug under the drone, if the field of view covered by the optical sensor includes enough area that's NOT moving, like all the interlock brick in my driveway, then it knows enough to ignore the motion of the small patterned rug moving within its sight. But if the motion of the rug fills a sufficiently large portion of the sensor's field of view, then the drone will stay "stationary" over the rug, following it as it's being pulled away, while ignoring contradicting IMU and GPS info.

"Low over water" just means the optical sensor will see more of the moving water and less of the static rocks etc. Or if the drone is hovering over a perfectly smooth mirror surface, the optical sensor will see only the reflection of the drone itself and end up following itself while thinking it's not moving at all.

To view this content we will need your consent to set third party cookies.
For more detailed information, see our cookies page.

Accept third party cookies

 

[djules75]

OP here. Of course I am probably biased , but even if this is by design, then there should be much more explicit warnings about flying over rivers or bodies of water. Also having warnings like "GPS signal weak. Hovering unstable. Fly with caution” is clearly an understatement (even though it's the vision system issue and it's potentially not at all related to GPS as per @Zbip57 posts and videos). Unstable shouldn’t mean “By the way, there’s also a risk that even with full stick movements, you might not be able to move your drone in the direction you intend to”.

The manual should clearly state something like: “Flying over of near a moving body of water carries significant risks and you might loose control of your drone. If you choose to do so, you should fly in S (sports) mode and take full responsibility for your flight.”
(Not even sure about Sport mode because it still uses the downward vision system, but at least, I would have had more top speed to counteract the speed of the moving white waters below…)

I went through the DJI mini 3 pro manual, and those are the relevant sections I found. Nowhere is it said that you should completely avoid flying over water, just that you should be careful and mostly that the downward vision system might be impaired. It never says anything about the drone fighting your commands as a result of this. Also, nobody in the DJI forum naturally guessed what happened to my drone when looking at my initial posting before the analysis from @BudWalker .

Let me know what you think!

Excerpts from DJI mini 3 pro manual below:

 

[Yorkshire_Pud]

Unfortunately I think DJI would be damned whatever they do about warnings in the app.
Why ?
Well written warnings would take time to read and expansive warnings would take up a lot of screen space. The pilot may not have that time to play with and DJI would be criticised for the screen space consumed.
Besides strictly speaking the pilot is meant to by flying the drone "eyes on the drone", not staring at the screen.
I don't think it is realistic to expect DJI to cover every scenario in the manual, the manual would need to be a full sized book to do that.

I would also suggest that more expansive warnings in the manual would be skipped over or skimmed and that their significance missed, especially by newbies.
Longer more detailed warnings in the manual would also, and obviously, lengthen the manual and I suspect increase the likely hood of the manual not being read thoroughly.

This is why I advocate practice sessions in combination with re reading the manual, nothing drives a point home like experiencing a behaviour for real, especially if the observed behaviour is unexpected.

Case in point RTH behaviour and the newer models of drone, I recollect that some of the newer RTH sections run to more than 6 pages and after reading many of the "I had problems with an RTH" threads it seems apparent that the problem actually lay in the pilot being unaware of the precise nature of the given RTH behaviour, especially where options combine to change the behaviour.
I have been caught out by that myself on a drone that has a relatively simple RTH behaviour.
Much of the vocabulary is new to a newbie and, as a newbie, I found that although I read the manual carefully before I first flew the relevant drone I did miss the significance of phrases etc. but, with an 'incident' behind me, I found that the manual described the resulting behaviour/risk, perhaps not well but it did describe it.

As a newbie only one thing caught my eye and made me think " I wonder what that means ?" and its subject is not relevant to this thread but I did investigate it.

Overall I think you flew a risky though very tempting location, unfortunately this time it turned around and bit you.


However I do suggest that you try a warranty claim but do that 'via' the DJI forum where the mods sometimes re-refer claims that are possibly open to dispute.

 

[djules75]

Unfortunately I think DJI would be damned whatever they do about warnings in the app.
Why ?
Well written warnings would take time to read and expansive warnings would take up a lot of screen space. The pilot may not have that time to play with and DJI would be criticised for the screen space consumed.
Besides strictly speaking the pilot is meant to by flying the drone "eyes on the drone", not staring at the screen.
I don't think it is realistic to expect DJI to cover every scenario in the manual, the manual would need to be a full sized book to do that.

I would also suggest that more expansive warnings in the manual would be skipped over or skimmed and that their significance missed, especially by newbies.
Longer more detailed warnings in the manual would also, and obviously, lengthen the manual and I suspect increase the likely hood of the manual not being read thoroughly.

This is why I advocate practice sessions in combination with re reading the manual, nothing drives a point home like experiencing a behaviour for real, especially if the observed behaviour is unexpected.

Case in point RTH behaviour and the newer models of drone, I recollect that some of the newer RTH sections run to more than 6 pages and after reading many of the "I had problems with an RTH" threads it seems apparent that the problem actually lay in the pilot being unaware of the precise nature of the given RTH behaviour, especially where options combine to change the behaviour.
I have been caught out by that myself on a drone that has a relatively simple RTH behaviour.
Much of the vocabulary is new to a newbie and, as a newbie, I found that although I read the manual carefully before I first flew the relevant drone I did miss the significance of phrases etc. but, with an 'incident' behind me, I found that the manual described the resulting behaviour/risk, perhaps not well but it did describe it.

As a newbie only one thing caught my eye and made me think " I wonder what that means ?" and its subject is not relevant to this thread but I did investigate it.

Overall I think you flew a risky though very tempting location, unfortunately this time it turned around and bit you.


However I do suggest that you try a warranty claim.

I understand and you are make some fair points. The manual cannot possibly cover all situations. But I think flying over water would warrant a bit extra caution than the few lines in the manual. The manual focuses on landing and vertical positioning issues when above water. I am not advocating for constant warnings but in my case "Hovering unstable" was an understatement. Nowhere did I read in the manual and on forums/YouTube videos before this about dangers of your drone actively drifting because of bad data fed to the Flight Controller (I wish I had seen the YouTube video I posted above before...)

"This is why I advocate practice sessions in combination with re reading the manual, nothing drives a point home like experiencing a behavior for real, especially if the observed behavior is unexpected." I am wondering which practice sessions would have prepared me for this? My point is that even if I had fully memorized the manual and done extensive training, that would not have prepared me for what I ran into, except if your practice locations managed to reproduce this exact situation by pure accident... Btw, I'm not a "PRO" DJI pilot but I have already 300+ km of flying my DJI mini 3 pro and have been flying FPV for 1 and half years.

Finally, I'm not sure I'll go with you with this "Besides strictly speaking the pilot is meant to by flying the drone "eyes on the drone", not staring at the screen.", because even if I had been closer with eyes on the drone, I doubt that would have helped a lot here. But point taken, even though realistically, most DJI pilots look at the screen more than at the drone.

Anyhow, my drone is more than a year old and I'm not under the DJI care program, so I'm not really expecting anything from a warranty claim... But I'll let you know when I hear back from them. Thanks for the discussion.

 

[BudWalker]

OP here. Of course I am probably biased , but even if this is by design, then there should be much more explicit warnings about flying over rivers or bodies of water. Also having warnings like "GPS signal weak. Hovering unstable. Fly with caution” is clearly an understatement (even though it's the vision system issue and it's potentially not at all related to GPS as per @Zbip57 posts and videos). Unstable shouldn’t mean “By the way, there’s also a risk that even with full stick movements, you might not be able to move your drone in the direction you intend to”.

The manual should clearly state something like: “Flying over of near a moving body of water carries significant risks and you might loose control of your drone. If you choose to do so, you should fly in S (sports) mode and take full responsibility for your flight.”
(Not even sure about Sport mode because it still uses the downward vision system, but at least, I would have had more top speed to counteract the speed of the moving white waters below…)

I went through the DJI mini 3 pro manual, and those are the relevant sections I found. Nowhere is it said that you should completely avoid flying over water, just that you should be careful and mostly that the downward vision system might be impaired. It never says anything about the drone fighting your commands as a result of this. Also, nobody in the DJI forum naturally guessed what happened to my drone when looking at my initial posting before the analysis from @BudWalker .

Let me know what you think!

Excerpts from DJI mini 3 pro manual below:

View attachment 177885
View attachment 177886

View attachment 177887

View attachment 177888

I agree 100%. Unfortunately, that’s not going help much. You might try posting this over on the DJI forum. At least there will be the possibility that DJI will read it before ignoring it.

 

[Zbip57]

Unstable shouldn’t mean “By the way, there’s also a risk that even with full stick movements, you might not be able to move your drone in the direction you intend to”.

I'm convinced that's not what happened here. As BudWalker pointed out in post #7 above, your stick inputs actually did have an effect.

The problem occurred during the period when you had the sticks centred with the drone in hover, and the optical sensor started tracking the swiftly moving river current.

I just mentioned this in another thread, that when the drone is at a longer distance away from you it might be easy enough to spot movement to the left/right or up/down, but it's really difficult to visually judge distance. So, while you were distracted switching between photo/video mode, you might not even have realized how quickly the drone was moving away from you when you had expected it to be staying in one spot while hovering.

It picked up significant speed, drifting backwards, while it tracked the rapid river current during hover. When you did give it full forward stick input, it did slow its rearward pace and would have started coming back to you. The left yaw input, while it was still moving backwards, is what started that curve toward the rock wall.

From the video, it looks like the drone had already stopped its rearward motion as it doesn't appear to moving forward or backward, just drifting laterally to the right into the rock wall. Then your full-left stick input unfortunately came a fraction of a second too late. You very nearly almost saved it there.

 

[Zbip57]

Nowhere is it said that you should completely avoid flying over water, just that you should be careful and mostly that the downward vision system might be impaired.

There have been many discussion threads with people sharing all sorts of, often misguided, opinions on what that actually means, "impaired".

It's important to distinguish between the two downward facing systems, the optical sensor and the infrared sensors. The optical sensor is what caused your problem, as it can track moving patterns when the drone is left to hover. The infrared sensors (or ultra-sonic sensors on older models) are purely for measuring height.

Although the infrared sensor is very accurate in measuring height above surfaces, up to the limit of its range (~30m?), and that number is recorded as VPS Height in the flight logs, it's not clear whether it ever plays any active role other than when landing, or within 0.5m (~2ft) of the ground.

I did a series of 4 videos demonstrating that in this playlist.

My Mini will not fly closer (lower) than 0.5m (~2ft) to the ground. The VPS system resists going any lower than that. I can hold the throttle slightly down, but the drone refuses to go any lower. If I hold the throttle all the way down, that triggers Auto-Landing and the drone will smoothly settle to an automated landing.

There are some surfaces that won't reliably reflect that infrared signal, so the VPS system cannot accurately sense its height. The signal can be absorbed by a deep shag carpet, or possibly by water. In that case, nothing prevents the drone from descending closer than 0.5m (~2ft) to the surface, and nothing prevents the pilot from dunking his drone into a lake.

Some people will advise to always disable the VPS system if you plan to fly over water. But, my understanding is that it can never actually cause a drone to fall into the water. When "impaired", the VPS height sensors can only ever fail to prevent a pilot from inadvertently flying his drone into the ground or water.

In my video where I was dragging the rug under my Mini, at 5:55 I had the downward sensors disabled when I landed the drone on my hand. I'm used to the drone settling slowly onto my hand. But, of course, with the sensors disabled the drone fell like a stone when I pulled the throttle stick down! D'oh!

 

[djules75]

I'm convinced that's not what happened here. As BudWalker pointed out in post #7 above, your stick inputs actually did have an effect.

I am not denying that actually. I've seen the analysis and the drone indeed slowed down and I agree with your summary. I think my only way out within that time frame would have been to immediately switch to Sports mode, but 4s is a short time when you don't understand what's happening.

"unable move your drone in the direction you intend to" was the perceived effect to me, the pilot. Unable to go where I wanted. But there is probably a better and more precise way to put it.

The problem occurred during the period when you had the sticks centred with the drone in hover, and the optical sensor started tracking the swiftly moving river current.

I just mentioned this in another thread, that when the drone is at a longer distance away from you it might be easy enough to spot movement to the left/right or up/down, but it's really difficult to visually judge distance. So, while you were distracted switching between photo/video mode, you might not even have realized how quickly the drone was moving away from you when you had expected it to be staying in one spot while hovering.

I also agree I very likely took more time noticing the drift because I was busy switching recording mode.

It picked up significant speed, drifting backwards, while it tracked the rapid river current during hover. When you did give it full forward stick input, it did slow its rearward pace and would have started coming back to you. The left yaw input, while it was still moving backwards, is what started that curve toward the rock wall.

From the video, it looks like the drone had already stopped its rearward motion as it doesn't appear to moving forward or backward, just drifting laterally to the right into the rock wall. Then your full-left stick input unfortunately came a fraction of a second too late. You very nearly almost saved it there.

In the end, my main point and all I am saying is that what happened to me is completely counter intuitive and that unless you've read about it or experienced it first hand, there is no way you can be prepared for it. Hence my recommendation for a better warning in the documentation. Heck, even someone like Ian in London got it wrong initially!

Happy flying everyone and I hope this thread will save a few drones in the future ;-).