With the introduction of the Mavic Mini there has been a noticeable uptick in the number of posts requesting help with lost or crashed aircraft. While there is plenty of help and advice available on this forum, and detailed guidance and options on several other websites, I thought that perhaps a single summary of the subject might be useful. If it does prove useful then I'll try to keep it updated, either in this post or followup posts, and perhaps other forum members will contribute additional thoughts on the subject to address errors and omissions.
1. INTRODUCTION
Starting back with the
Phantom 4, DJI aircraft and control software record an impressive amount of flight data to various log files both on the aircraft and on the mobile control device (phone, tablet etc.) which is extremely useful in determining the cause and outcome of unexpected events. These log files contain far more data than the simple flight summary files that appear in the DJI GO 4 app, but are not as simple to access. Most of them are encrypted, but many can be read using widely available software or web-based converters.
Before discussing the log files themselves it's possibly worth a brief description of how these flight control systems work. Skip to section 2 if you are not interested in this level of detail. I've included some links for optional further reading.
Ignoring fully manual mode, which is only available by
changing a firmware parameter, aircraft flight is controlled entirely under the hood by the onboard flight controller (FC). It converts stick input commands to move up/down, forwards/backwards, left/right and CW/CCW into motor adjustments to achieve the requested movement. Sticks centered means hold position, altitude and heading (P-GPS mode) or hold altitude and heading with pitch and roll set to zero (ATTI mode). Stick inputs simply request specific movement relative to those baselines.
Exactly how these flight control systems work is a rather deep and arcane subject. At the simplest level, the aircraft have either one or two
MEMS strapdown IMU packages that comprise 3-axis accelerometers and rate gyros, together with 3-axis magnetometers (compass), a barometric sensor, and a GNSS module that receives GPS and GLONASS signals. Anyone with a cursory familiarity with basic mathematics will note that there is, therefore, redundancy in the flight data. In theory, given an initial position and heading, transformation of the accelerometer and rate gyro data into the earth frame of reference followed by integration with respect to time yields velocity and displacement as a function of time - pure inertial navigation. In practice it doesn't work with these kinds of IMUs, due to bias (zero offset errors) and drift (small changes in sensor gain), which lead to errors in the integrated quantities. The result, which is easy to replicate since the raw sensor data are available, is that after just a few tens of seconds of flight the inertial solution ends up noticeably incorrect.
However, the accelerometer and rate gyro data are, in fact, the only data measured fast enough to provide responsive aircraft control, and thus they are the primary data source for the control algorithm. Once the aircraft yaw (heading) is initialized by the compass, the aircraft attitude, velocity, position and height are computed primarily inertially. The IMU sensor drift and bias issues are corrected more gradually using the GNSS, compass and barometer data by a
sensor fusion algorithm, most likely a
Kalman filter or equivalent, keeping the absolute velocity, position, heading and altitude correct within the accuracy of those sensors. The aircraft attitude is actually not tracked as the derived
Tait–Bryan angles (pitch, roll and yaw) since that description becomes indeterminate for certain orientations. Instead the FC uses
rotation quaternions which, while a more obscure branch of mathematics, has a number of advantages.
Most flight control problems are caused by:
- Loss of GNSS positioning, which will generally lead to the FC switching to ATTI mode unless it has active VPS positioning available;
- Disagreement between the IMU and compass yaw values, often caused by taking off from a locally magnetically distorted site;
- Wind speeds in excess of the design limits of the aircraft;
- Lost/damaged props or motor problems, leading to loss of propulsion;
- Power interruption due to battery disconnects;
- FC/IMU processor issues.
Mobile device DJI TXT logs
These logs are created by the standard
DJI SDK and run from motor start to motor stop (or signal loss), recording telemetry transmitted from the aircraft to the RC at 10 Hz in a few hundred data fields. These logs don't include any raw sensor data - only the IMU solutions that are the output from the sensor fusion algorithm. They also include battery data, gimbal and camera data, flight status and error flags, but no motor data.
They are often all that is needed for analysis of simple events, but the lack of raw sensor data makes it difficult to identify exact causes if there are IMU problems, and the lack of motor data is a problem when there are propulsion issues. Additionally, the relatively slow data rate and a couple of tenths of a second latency means that events rapidly followed by aircraft shutdown (e.g. many crashes) are not recorded.
Aircraft DAT files
These contain the most comprehensive data, logged at the highest rates. A DAT file is started at aircraft power up and continued until power down, and includes numerous boot sequence, sensor calibration and diagnostic data in the flight event stream. There are also hundreds of data fields, many of which are flags or diagnostic computations of unknown types, probably understood only by DJI. However, among these are the raw and processed sensor data and the IMU solution for aircraft attitude, position, velocity and heading, together with battery and motor data, recorded at rates varying from 5 Hz to 200 Hz. These data are very valuable for diagnosing flight control problems. The files exist independent of the mobile device control app being used.
Unfortunately, on recent DJI models (Mavic Air,
Mavic 2 and Mavic Mini) the decryption keys are hidden, and so those are not readable except by DJI.
Mobile device DAT files (DJI GO 4 app)
These contain a subset of the aircraft DAT files, including most of the raw sensor data, recorded at a lower data rate of 10 Hz. They are also started at aircraft power up and closed at aircraft shutdown, provided that the app is connected - otherwise it will be the subset of that duration when the app is connected. Not as good as the aircraft DAT files, but nearly as good for most needs, and always available if the control app is DJI GO 4. DAT files are not created when using Litchi or other third party control apps.
Mobile device third-party control app log files
Third-party control apps, such as Litchi, create their own custom log files in various formats using the DJI SDK, generally in CSV format. None that I have seen are as comprehensive as the DJI TXT log, let alone the DJI DAT log.
Since Litchi is probably the most common third-party app, it's worth noting that, under iOS and Android, Litchi creates its own CSV log and a standard DJI TXT log in a separate directory. No DAT file though.
3. LOG FILE RETRIEVAL
Mobile device DJI TXT logs
The log naming convention, based on the date and time of the start of the flight, is: DJIFlightRecord_YYYY_MM_DD_[hr-min-sec].txt.
With iOS devices running DJI GO 4 you need to access the app files via computer, either using iTunes or with a file system browser such as
iExplorer. The TXT logs are in
Apps » DJI GO 4 » FlightRecords. The same method can be used to access DJI Fly files in
DJI Fly » FlightRecords, but with DJI Fly they are also accessible using the iOS Files app by selecting
Browse » On My iPhone » DJI Fly » FlightRecords.
With Android devices the file system should mount when plugged into a Windows machine, or via various software options on a Mac. With DJI GO 4 the TXT logs are in DJI » dji.go.v4 » FlightRecord. With DJI Fly they are in DJI » dji.go.v5 » FlightRecord.
If you want help analyzing the data then you can either retrieve the logs and post them here directly, or upload them to
PhantomHelp or
AirData and post the link back here.
AirData also has
automatic log upload (sync) options directly from most of the common control apps.
Aircraft DAT files
These are only readable from the Mavic Pro and Mavic Pro Platinum (plus Phantom 3, Inspire 1,
Phantom 4,
Phantom 4 Pro,
Inspire 2,
Inspire 2 Pro, Matrice 100, Matrice 200, Matrice 600, and Spark). A guide to retrieving them from the aircraft can be found on
@BudWalker's
DatCon website. They are much larger files than the mobile device files, and generally too large to post directly to this forum. The best solution is to upload them to Dropbox or similar and then post a link.
Mobile device DAT files (DJI GO 4 & DJI Fly)
The DAT file naming convention, based on the date and time of the start of the file, is: YY-MM-DD-hr-min-sec_FLYXXX.DAT., where XXX is the flight recorder file index from the HOME_dataRecorderFileIndex field in the txt log.
These are retrieved by the same method as the TXT logs. Under both iOS and Android they are in a subfolder, MCDatFlightRecords, in the folder that contains the TXT logs. In some cases, for reasons not fully explained but possibly mobile-device hardware related, and most often under Android, DAT files are not created and that folder is empty. Uninstalling and reinstalling the app sometimes fixes that. One user also found that the process required manual deletion of the app folder (apparently not deleted automatically in the uninstall process) before reinstalling the app.
The DJI Fly app appears to delete the DAT files if and when it syncs flight records with the DJI servers, and so they are often missing from the MCDatFlightRecords folder.
Litchi log files
On iOS devices, the Litchi CSV logs are in Litchi » Documents » flightlogs, and the DJI TXT logs are in Litchi » Documents » SDK_logs » FlightRecord.
On Android devices the Litchi CSV logs are in LitchiApp » flightlogs. Litchi also creates a standard DJI TXT log, which for some reason it stores in a subdirectory in the DJI app directory, DJI » com.aryuthere.visionplus » FlightRecord.
In the case of automated Litchi missions, it's also important to post a link to the flight mission profile.
4. LOG FILE CONVERSION AND VISUALIZATION
There are various options to visualize and examine the contents of the log files, depending on type. Some of the commonly used ones are described below:
DJI TXT logs
The
PhantomHelp website will display a summary of the flight with some of the telemetry. You can also download the original TXT log and the full file contents in CSV format. It doesn't provide graphical analysis, but you can open the CSV file in Excel or dedicated data analysis programs.
The
AirData website will also read TXT logs, giving a different view of the data. It has a useful condensed view of app notifications, and a fairly advanced analysis of the computed wind field during the flight (subscription only). It doesn't provide a full downloadable version of the log - only a limited subset of the data. It also, by default, allows download of the original TXT log.
The Windows version of
CsvView will convert DJI TXT logs, and provides numerous powerful options to visualize the flight data.
TXTlogToCSVtool is another offline Windows program that reads and converts TXT logs to CSV format. In fact the PhantomHelp website and CsvView both use this to perform their conversions. There is now a separate version for Mavic Mini logs. There are no visualization options - you simply get the full CSV version of the log that you can open in other data analysis programs.
Aircraft DAT files
Phantom 4 and Mavic Pro DAT files can be converted to CSV format using
CsvView or
DatCon, which is a Java application that runs under OS X and Windows. These files are recorded at rates up to 200 Hz for some of the data, and contain hundreds of data fields. They are extremely data rich, but not for the faint of heart when it comes to trying to interpret them.
@BudWalker is constantly updating the parsing engine, but a partial description of some of the more basic data fields is on his
website.
Mobile device DAT files (DJI GO 4 app)
These are converted to readable CSV files using
CsvView or
DatCon, and can be visualized in
CsvView under OS X and Windows. They are recorded at just 10 Hz and are much smaller than the aircraft DAT files, but contain many of the same fields.
AirData also reads mobile device DAT files and displays a similar set of data as it does with the TXT logs.
Litchi log files
These logs are recorded as CSV files and so they don't need any conversion.
PhantomHelp and
AirData will display them, or they can be opened by Excel or data analysis software.
5. LOG FORENSICS
Any attempt at a description of this subject is going to be very superficial - at this point I could write a short book on telemetry data analysis. Over the past few years I've developed a few streamlined approaches to looking at the data, depending on the type of event (crash, flyaway, loss of control, unexpected behavior etc.), but most of them involve significant data processing in the form of frame of reference transformations, quaternion algebra and some calculus. Those all require a significant data analysis package - I use
Wavemetrics Igor Pro for which I've written a number of data processing functions. Igor Pro is particularly suited to this kind of work, and since user functions are compiled, rather than interpreted, it chews through huge datasets very fast. To give a few examples:
- In cases of flight control issues, I look for discrepancies between the IMU sensor fusion solutions for position and velocity, and check for appropriate correlations between the stick inputs, motor speeds (DAT files only), aircraft attitude, and horizontal velocities and accelerations. Example.
- With flyaways (or generally blowaways) in P-GPS the most effective technique is to extrapolate the battery level and solve for the intersection with the autoland, and then extrapolate the aircraft position with drift velocity to determine where it should autoland. Here's a detailed description of that process in practice.
- For power loss or FC failure leading to motor shutdown I use an aerodynamic aircraft model together with the computed wind field as a function of height to run a drag-dominated finite-difference numerical solution for the descent path of the aircraft, writing to a KML descent track file. Example.
That said, there are various simple checks that can be performed by looking at the telemetry. The data field names below are from the DJI TXT log, but there are equivalent fields in the DAT files. Some are text fields, while others are numerical.
- OSD.flyTime – elapsed flight time;
- OSD.flycState – the flight control mode (P-GPS, ATTI, RTH, autoland etc.)
- SMART_BATTERY.battery – the battery level in percent;
- SMART_BATTERY.goHomeBattery – the computed battery percentage that triggers RTH, based on distance from the home point;
- SMART_BATTERY.landBattery – the computed battery percentage that triggers autolanding, based on height above the home point;
- OSD.pitch, OSD.roll, OSD.yaw – the aircraft attitude as pitch, roll and yaw (IMU sensor fusion yaw solution - not the compass heading);
- OSD.height – height above the home point;
- OSD.sWaveHeight – VPS height above the ground;
- OSD.gpsNum – number of GNSS satellites locked;
- OSD.gpsLevel – navigation health level (1 - 5), which is a measure of confidence in the sensor fusion solution;
- OSD.isGPSused – is GNSS used for positioning;
- OSD.isVisionUsed – is VPS (rather than GNSS) being used for positioning;
- RC.aileron, RC.elevator, RC.rudder, RC.throttle – the recorded stick inputs;
- OSD.latitude, OSD.longitude – the sensor fusion solution for position (not the raw GNSS position data, which is only in the DAT file);
- OSD.xSpeed, OSD.ySpeed, OSD.zSpeed – velocity north, east and down, respectively;
- MC_PARAM.failSafeAction – the failsafe action (RTH, hover or land).
That's just a partial list of some of the most useful data fields when it comes to trying to identify flight problems.
In all but the most simple cases, I recommend posting to the forum a description of the flight together with both the DJI TXT log and mobile device DAT log. There are several forum members who have experience in interpreting these files by various methods. For those who might be interested, the best tutorials for understanding and replicating the analysis methods are probably the forensic threads on this forum, and any questions for further explanation are always encouraged.
Comments welcomed...
EDITS:
12/26/19: Added the full range of DJI aircraft with readable onboard DAT files.
12/26/19: Added the location of Litchi DJI SDK TXT log files.
12/27/19: Added explicit retrieval directions for DJI Fly.
02/10/20: Added that DJI Fly auto deletes DAT files.
