Litchi flight: why is telemetry flight path different from real flight path?

[marklyn]

I could use some de-confusing here.
I'm puzzled why my Litchi flight data (flight path) differs from my actual flight path (evidenced by video).
I set up a 6 minute Litchi mission.
The Mavic did lose signal for a couple of minutes but that's happened before on a number of missions and when I look at the flight path on Airdata UAV (or download the Litchi flight logs) the flight path always matched the flight path from the video recorded. Today it didn't.
The flight path screenshot shows the programmed flight path (yellow), the actual flight path flown by evidence of the video (green) and the incorrect flight path that was *not* flown (red).
Hoping someone can shed some light on this. Also included the Litchi flight log (zipped).

 

[Logger]

The yellow flight path is actually the Litchi Flight Log as recorded on your mobile device. It is not your programmed mission.
Your log shows that over 4 of the 10 minutes was flown with the down link between the aircraft and the Litchi app on your mobile device lost.


At points J, O and P the periods were quite long. The Litchi app and the associated log has no way of knowing where the aircraft is during this period and as a result deduces & depicts a straight line between points omitting the actual flight for the blackout periods. If you were to pull the log from the aircraft you would see the actual flight path.

Your red line more or less represents the period where the signal was lost and the Litchi app had no idea where the AC was as it flew the mission along the green line.

So whenever the aircraft is disconnected from the RC during a litchi mission the subsequent log will not reflect the correct flight path for this period. Sometimes it may look like it has recorded properly during signal loss - if this happens to occur during a programmed straight leg. But this is just an illusion.

 

[marklyn]

Thanks for looking at that logger.
So, I guess my question is this...
Why, out of 200 flights (223 flight miles), I've never noticed this before?
There have been many flights where I've lost signal for 1+ minutes but when the telemetry is uploaded to AirData UAV, it matches what I see on video in the flight path.
It sounds like if I wanted to see the true/accurate flight path, that might reside on the Mavic in one of the unusable black box files?
Any insight on this sure would help me understand.

 

[Logger]

...So, I guess my question is this...
Why, out of 200 flights (223 flight miles), I've never noticed this before?
There have been many flights where I've lost signal for 1+ minutes but when the telemetry is uploaded to AirData UAV, it matches what I see on video in the flight path.

Most probably because you have not looked closely enough at the litchi log data. For instance in your flight above you have pointed out the seeming inconsistency that it mapped the red line when it actually flew the green path. I see that as the 138 seconds up until Warning O. However that was just one of the 3 segments. You did not notice the other two in the Litchi path log. I only had a cursory look at your data and I surmise that the 52 second loss to Warning J was the last outbound straight path - from just SE of the road to fartherst point. Then the 72 sec loss up to warning P was the second last straight line depicted showing a SE direction along the north side of the road. As soon as it regained signal it received and followed your RTH command. Hence the final vector home. But the other two were probably close enough to your planned Litchi mission that you did not notice?



Here are two screen grabs of a Litchi mission I do around my farm all the time. It is 9km long. The red circles depict two LOS regions. In the second image I bothered to locate myself and the RC where I could retain signal throughout and the litchi log depicts the true path because the telemetry was continuous.

It sounds like if I wanted to see the true/accurate flight path, that might reside on the Mavic in one of the unusable black box files?
Any insight on this sure would help me understand.

DJI Assistant 2 software will allow you to access the log from the aircraft. Otherwise simply put locate yourself where you will not lose signal.

 

[Kilrah]

Why, out of 200 flights (223 flight miles), I've never noticed this before?

Simply becasue it's the first time the drop was at a time where the aircraft was flying a "complex" path and not a mostly straight line.

 

[marklyn]

Thanks Logger, you're right of course. I glanced at another mission that was lengthy and over hills and you're correct, It was never obvious to me. Early on when I flew I didn't use the telemetry data to make videos with tracking maps as I've been doing for the last 75 or so missions so I didn't notice.
Wow. It would be nice for a future version of Mavic to store a "real" copy of that data on the craft and make it available to copy down.
I've glanced at the flight data files from the Mavic but they don't look like they are very useful (or easy to use) for obtaining flight telemetry data for post video production.

 

[sar104]

Thanks Logger, you're right of course. I glanced at another mission that was lengthy and over hills and you're correct, It was never obvious to me. Early on when I flew I didn't use the telemetry data to make videos with tracking maps as I've been doing for the last 75 or so missions so I didn't notice.
Wow. It would be nice for a future version of Mavic to store a "real" copy of that data on the craft and make it available to copy down.
I've glanced at the flight data files from the Mavic but they don't look like they are very useful (or easy to use) for obtaining flight telemetry data for post video production.

The aircraft records the full record in a DAT file, which previously was easy to access. More recent Mavics (Pro and Air) are storing some or all of the DAT files in locations that are not user accessible. You can check to see if yours are retrievable:

How to retrieve a V3 .DAT File from the AC

 

[marklyn]

Logger, I've gone to that site several times over the past year and have downloaded (and extracted .dat files) from the Mavic but I've not ever seen instructions on how to use that telemetry data in post flight apps like RaceRender or Dashware to overlay flight info onto the video. The csvview and datcon apps seem to allow a very seasoned person the ability to chart out different flight characteristics or rather to pinpoint and review specific fight info in great detail. I know this is how some of the well seasoned pilots can take apart a flight to tell all of us what went wrong in a crash, etc.
Also, I've never been able to understand the extracted flyxxx.dat files even the converted flyxxx.csv files. The extracted flyxxx.dat files don't correlate with actual flights in a way that I understand how to find the flight data I'd like to see. If anyone knows of a tutorial on how to extract flight telemetry data from these files on specific flights, I'd sure like to see that. I would think that would be more accurate than anything that is transmitted back to the DJI Go/Litchi/etc. app tethered to the controller. Right?
Still, I will continue to look at the flyxxx.dat files and continue to read what I can find. I haven't pulled the .dat files from my Mavic in about 3 months. DJI seems to be very guarded in what they will ( and mostly won't) tell you about how to use/access these files to their fullest extent.

 

[sar104]

Logger, I've gone to that site several times over the past year and have downloaded (and extracted .dat files) from the Mavic but I've not ever seen instructions on how to use that telemetry data in post flight apps like RaceRender or Dashware to overlay flight info onto the video. The csvview and datcon apps seem to allow a very seasoned person the ability to chart out different flight characteristics or rather to pinpoint and review specific fight info in great detail. I know this is how some of the well seasoned pilots can take apart a flight to tell all of us what went wrong in a crash, etc.
Also, I've never been able to understand the extracted flyxxx.dat files even the converted flyxxx.csv files. The extracted flyxxx.dat files don't correlate with actual flights in a way that I understand how to find the flight data I'd like to see. If anyone knows of a tutorial on how to extract flight telemetry data from these files on specific flights, I'd sure like to see that. I would think that would be more accurate than anything that is transmitted back to the DJI Go/Litchi/etc. app tethered to the controller. Right?
Still, I will continue to look at the flyxxx.dat files and continue to read what I can find. I haven't pulled the .dat files from my Mavic in about 3 months. DJI seems to be very guarded in what they will ( and mostly won't) tell you about how to use/access these files to their fullest extent.

If you can get the logs then the data are all there in easily accessible form via the converted csv files. How to use those data in terms of importing it into other programs is a different question, but there is no mystery in reading the logs. If you can describe the data that you need or the destination program's requirements then it should be easy to figure out.

 

[marklyn]

I actually have one of the dat files that I used datcon to convert into a csv file to play around with. The headers of the csv file seem more foreign to me than a standard litchi flight (csv) file or an exported airdata UAV csv file. Those two file formats and header info are mostly understandable.
Some of the data headers in the converted dat to csv file appear to be duplicated somewhat with different names (ie: IMU_ATTI(0):Longitude vs GPS(0):Long) as one example.
I'm trying to get some basic telemetry data like the data used for a tracking map, altitude, speed, gimbal angle, drone direction.
I also don't understand the extraction of the .dat files, the file(names) don't seem to correlate with dates/times I've flown.
I'm still looking into it and re-reading some information I've come across before but if anyone has already done something similar, I'd appreciate any shortcuts offered.
My goal is to use this data in RaceRender for telemetry overlays of the video.
I'm going to try to convert another dat file, which is much larger (around 300mb) to see if that has more/better data in it.

 

[marklyn]

OK, so I quickly stumbled on something positive.
I had previously used a couple of flyxxx.dat files that were small in size so when I converted them the KML didn't work in GE. The CSV files data looked wonky so now I assume these files were created when just turning on the Mavic, with no flight performed.
When I look at the much larger files, say 100mb or larger, and convert them to kml, csv, etc., then GE loads the file and shows the flight plan in completeness and the csv files look a LOT better.
I guess that I never realized the large files were actually the files created from the drone being on a long time (ie: actual flights) whereas the smaller files were when I probably turned on the Mavic to do something other than a flight.
Assuming I'm correct (please tell me if I'm on the right track), then how often would I need to pull the dat files before they are overwritten?

Thanks!!!!

 

[Kilrah]

That's correct, DATs are stored ANY time the aircraft is powered on, one file per power cycle and if doing multiple flights without powering off they will all be in the same file. You should take them out every 3-4 batteries maybe, testing might be needed.

 

[sar104]

I actually have one of the dat files that I used datcon to convert into a csv file to play around with. The headers of the csv file seem more foreign to me than a standard litchi flight (csv) file or an exported airdata UAV csv file. Those two file formats and header info are mostly understandable.
Some of the data headers in the converted dat to csv file appear to be duplicated somewhat with different names (ie: IMU_ATTI(0):Longitude vs GPS(0):Long) as one example.
I'm trying to get some basic telemetry data like the data used for a tracking map, altitude, speed, gimbal angle, drone direction.
I also don't understand the extraction of the .dat files, the file(names) don't seem to correlate with dates/times I've flown.
I'm still looking into it and re-reading some information I've come across before but if anyone has already done something similar, I'd appreciate any shortcuts offered.
My goal is to use this data in RaceRender for telemetry overlays of the video.
I'm going to try to convert another dat file, which is much larger (around 300mb) to see if that has more/better data in it.

Those fields are not actual duplicates. One is the IMU-computed location, calculated using the accelerometer and rate gyro data and adjusted at low gain by the GPS location data, while the other is the raw GPS coordinates. While they should be the same, the former is updated at a much higher frequency and they can diverge if the sensor fusion scheme fails.

 

[BudWalker]

I actually have one of the dat files that I used datcon to convert into a csv file to play around with. The headers of the csv file seem more foreign to me than a standard litchi flight (csv) file or an exported airdata UAV csv file. Those two file formats and header info are mostly understandable.
Some of the data headers in the converted dat to csv file appear to be duplicated somewhat with different names (ie: IMU_ATTI(0):Longitude vs GPS(0):Long) as one example.
I'm trying to get some basic telemetry data like the data used for a tracking map, altitude, speed, gimbal angle, drone direction.
I also don't understand the extraction of the .dat files, the file(names) don't seem to correlate with dates/times I've flown.
I'm still looking into it and re-reading some information I've come across before but if anyone has already done something similar, I'd appreciate any shortcuts offered.
My goal is to use this data in RaceRender for telemetry overlays of the video.
I'm going to try to convert another dat file, which is much larger (around 300mb) to see if that has more/better data in it.

Yeah, the guy that developed CsvView/DatCon is way behind on the documentation. Caused partly by the fact that it was recently figured out how to see a lot more fields that now need documenting. For your example the GPS:Longitude comes from the GPS receiver and IMU:Longitude is computed from GPS:Longitude as well as IMU data.

You might want to look here for some existing column definitions
V3 .CSV column descriptions

And look here for hints on how to get the right .DAT
How to retrieve a V3 .DAT File from the AC

 

[marklyn]

That's correct, DATs are stored ANY time the aircraft is powered on, one file per power cycle and if doing multiple flights without powering off they will all be in the same file. You should take them out every 3-4 batteries maybe, testing might be needed.

That's kind of what I've been doing, even though I wasn't using or understanding them yet, I did a dump of the files every 3-4 months, HOWEVER, I have now noticed that there is no new .dat files, only old ones. I've done at least 20+ flights since I did the last .dat download to my pc. Did DJI turn off that feature? Or, did I maybe do something to turn off data logging on the Mavic???

 

[BudWalker]

That's kind of what I've been doing, even though I wasn't using or understanding them yet, I did a dump of the files every 3-4 months, HOWEVER, I have now noticed that there is no new .dat files, only old ones. I've done at least 20+ flights since I did the last .dat download to my pc. Did DJI turn off that feature? Or, did I maybe do something to turn off data logging on the Mavic???

Actually, this happens quite frequently to pilots using FW 01.03.0900 and after - myself included. As I understand it some .DATs get recorded on flash memory making them effectively non-existent to most people. I think there is a way of retrieving those .DATs but it requires a rooted Mavic and some advanced knowledge.

I've discovered how to keep .DATs from being recorded on the flash memory in the first place. Basically, the SD card is kept close to empty by periodically deleting the .DATs from the SD card. So far, all of my new flights get recorded on the SD card where I can get them. I've been doing this for a month or so and it has worked every time so far. My Mavic has been rooted but I don't think that matters.

Here's my method for deleting .DATs from the SD card.
1) Connect USB port to the PC and then start a FTP client (I use FileZilla).
2) connect to 192.168.42.2 with UserName = nouser, Password = nopass, port = 21
3) On the remote site (i.e., the Mavic) navigate to the /blackbox/flyctrl directory and delete the .DATs you don't want.
4) Navigate to the /blackbox/vision directory and delete some of the directories found there

Just to be clear this method can't retrieve a .DAT recorded on the flash memory (so it won't help here). It only makes it possible to retrieve future flights.

 

[marklyn]

Actually, this happens quite frequently to pilots using FW 01.03.0900 and after - myself included. As I understand it some .DATs get recorded on flash memory making them effectively non-existent to most people. I think there is a way of retrieving those .DATs but it requires a rooted Mavic and some advanced knowledge.

I've discovered how to keep .DATs from being recorded on the flash memory in the first place. Basically, the SD card is kept close to empty by periodically deleting the .DATs from the SD card. So far, all of my new flights get recorded on the SD card where I can get them. I've been doing this for a month or so and it has worked every time so far. My Mavic has been rooted but I don't think that matters.

Here's my method for deleting .DATs from the SD card.
1) Connect USB port to the PC and then start a FTP client (I use FileZilla).
2) connect to 192.168.42.2 with UserName = nouser, Password = nopass, port = 21
3) On the remote site (i.e., the Mavic) navigate to the /blackbox/flyctrl directory and delete the .DATs you don't want.
4) Navigate to the /blackbox/vision directory and delete some of the directories found there

Just to be clear this method can't retrieve a .DAT recorded on the flash memory (so it won't help here). It only makes it possible to retrieve future flights.

Think I just sent you a pm Bud, but thanks... I forget what are the vision files for?

 

[BudWalker]

Think I just sent you a pm Bud, but thanks... I forget what are the vision files for?

I don't know a lot about the vision files. When you use DJI Assistant they show up as being extracted at the same time the .DATs are being extracted.

DatCon doesn't know what to do with them and ExtractDJI ignores them. They take up space so they need to be removed for this scheme to work

 

[Logger]

@marklyn if you just want basic telemetry of your flightpath, why don't you just use the SRT ffile that the Mavic creates when you enable Video captions? The SRT files correlate to each video segment and would seemingly make your work flow much simpler. I have never done it myself, so I cannot speak first hand, but there is a discussion on using SRTs with Dashcam over here Mavic SRT files usage (Dashware, other) Perhaps the SRTs dont have the granularity for what you want, but they look OK to me. This was simply from telling the SRT viewer to open an SRT that was made with one of those Farm lap videos I mentioned earlier.



You need to open the camera settings in the DJIGo4 app and select Video Captions. It will then create the SRT files when you record with Litchi as well.

 

[marklyn]

Hrm. I didn't know anything about these SRT files. I will investigate this! Thanks