Something a little weird with the SD card

[KennethH]

Indeed, why reformat them at all?

Today is my DJI FPV assembly day! In anticipation of this blessed event, I bought two new SanDisk Extreme 256 GB microSD cards, which I plan to pop into the drone and the goggles with no preliminary other than taking them out of the packaging.

I can't imagine reformatting a card unless it had failed.



MMFPV!

I've learned that the problem I had with a ScanDisk card for the last three days was simply a bad card. I plugged a SanDisk Ultra 128 gb into the Mini 2 today, and it has been very happy. Did a lot of test footage that looks good (for what the subject matter is). From now on I'll download off the SD card, and then delete the video files with DJI Fly.

 

[sbunting]

If you want a bucket full of misinformation, start a discussion about SD cards and formatting. Generally, whether a MicroSD card is formatted on the device in which it is intended to function or on a computer shouldn't matter, but sometimes it does.

My business is digital forensics and I have tools and devices that require the media to be formatted with the tool or device, and for varying reasons. As a general rule, I find it better to format media in the device in which it was intended to work, as that device knows best which format to use.

DJI devices format 32 GB cards or smaller, in FAT32. DJI devices format 64 GB devices and larger in exFAT. FAT32 has a 4GB file size limit, whereas exFAT does not, so you don't have to be careful about file sizes on an exFAT formatted device. Not that file size is unlimited with exFAT (16 exabytes), but as a practical matter it is an insignificant consideration for drone use.

When you format on a computer, Windows adheres to the same scheme as does DJI, (32 GB or less defaults to FAT32 and 64GB or more defaults to exFAT). Yet, if you change the defaults to NTFS, DJI will see it immediately as an error. If you format on a Mac, it will default to HFS+ or APFS, depending on which version of OS X you are running. Neither of those will work on DJI (results in SD card error) and so you will have to change the defaults to use either FAT or exFAT.

DJI runs on a an embedded Linux OS and so it should see and work with ext2, 3, or 4, but it will not. I've tried formatting a card with ext4 (a fault-tolerant file system), one that is native to Linux and should work in a Linux environment, BUT DJI will not recognize it. The DJI programmers have done the same thing that Android does, which is to not recognize its own native file systems at the user interface level. They don't want to introduce incompatible file systems from Linux into a mostly Windows and Mac world, hence they stick with the universal formats of FAT and exFAT. They want to make life simple and reduce customer complaints.

Unfortunately, neither FAT nor exFAT are fault tolerant file systems, but DJI forces us to use those formats. They are NOT journaling file systems and can therefore corrupt easily during a power bump or an unclean dismount, which does happen. Even though Windows is supposed to allow you to just "yank out" removable media, you are best to always eject it cleanly (dismount), in both Windows and Mac. If for any reason you get a message you can't, it's busy and yanking the card while it is in use is almost a surefire way to cause an error that requires fixing or maybe even reformatting.

Whether a format erases all data can be answered with the word, depends. A quick format in Windows does not erase data, but only resets a few housekeeping files, in simple terms. A regular format, one in which the Quick Format box has been unchecked, in Windows overwrites data during formatting since Windows 7 came out. Mac does not overwrite data during formatting, but it can optionally. It is similar to a Windows quick format. Overwriting data, i.e. wiping, takes a lot of time, especially if it is a platter drive, but we don't have such in drones, only flash media, which is an entirely different beast altogether.

The memory cells in flash media have a finite number of times of reads and writes before they fail. It would not do if different portions of the card failed soon and others failed much later in life, so the manufacturers created proprietary wear-leveling algorithms that move the data around so that it wears out evenly. These algorithms are also 'aware' when files are deleted so they can more efficiently make use of space during wear-leveling. Thus deleting files normally doesn't delete the data, but the wear leveling algorithm kicks in and therefore deleted data gets overwritten fairly quickly. Just how quickly? Depends on lots of factors well beyond the scope of this conversation.

So basically the more a flash media card is used (reads / writes), the more quickly it will eventually fail. That said, doing a lot of formatting adds to the number of reads/writes and, in theory, wears the card out sooner. In practice, I have never worn out a flash memory card yet. I've lost them, drowned them, broken them, had them stolen, but more often than not I retire them and replace them with larger and faster media.

There are times when despite all the care, the card just won't read, usually on the computer. On the computer, I usually can remedy that by plugging into a different port or by rebooting. If that fails, I resort to a disk repair and as a last resort, I reformat. If I have to reformat, I try to rescue any data first. I find the Access Data's FTK imager will allow me to get the data when the computer can no longer mount the file system. It's free and if it can read it physically, you can copy your data off. Just search and download. There's no substitute for frequent data backups. Usually when you lose data, you haven't been diligent in that regard.

Just a few tips that will help you:
1) Format your media in the device for which it was intended, as it will choose the best format and underlying settings (not all formatting is equal)
2) Use good, up-do-date, card readers. Take care with cables and have spares. Swap out a reader or cable if you are having issues. Trying a different port or reboot.
3) ALWAYS do a clean dismount. Eject the media properly in either Windows or Mac. Failing to do so causes more issues than any other reason in my experience. Many people see me doing this in Windows and ask what I'm doing and why. Clearly they know little about properly dismounting media devices. Mac users seem to understand "dismounting" as they get scolded by the operating system if they fail to do so, whereas Windows could care less it seems. Along the same lines, don't remove the SD from the drone while powered on. Power off first to ensure a clean dismount.
4) Don't let other people use your media cards in their devices. They may have malware or they may not care to properly eject the media card.
5) Always backup your data. Minimize how much you can lose at any given point in time.

Best and enjoy flying...

Steve

 

[Eagle Eye 62]

If you want a bucket full of misinformation, start a discussion about SD cards and formatting. Generally, whether a MicroSD card is formatted on the device in which it is intended to function or on a computer shouldn't matter, but sometimes it does.

My business is digital forensics and I have tools and devices that require the media to be formatted with the tool or device, and for varying reasons. As a general rule, I find it better to format media in the device in which it was intended to work, as that device knows best which format to use.

DJI devices format 32 GB cards or smaller, in FAT32. DJI devices format 64 GB devices and larger in exFAT. FAT32 has a 4GB file size limit, whereas exFAT does not, so you don't have to be careful about file sizes on an exFAT formatted device. Not that file size is unlimited with exFAT (16 exabytes), but as a practical matter it is an insignificant consideration for drone use.

When you format on a computer, Windows adheres to the same scheme as does DJI, (32 GB or less defaults to FAT32 and 64GB or more defaults to exFAT). Yet, if you change the defaults to NTFS, DJI will see it immediately as an error. If you format on a Mac, it will default to HFS+ or APFS, depending on which version of OS X you are running. Neither of those will work on DJI (results in SD card error) and so you will have to change the defaults to use either FAT or exFAT.

DJI runs on a an embedded Linux OS and so it should see and work with ext2, 3, or 4, but it will not. I've tried formatting a card with ext4 (a fault-tolerant file system), one that is native to Linux and should work in a Linux environment, BUT DJI will not recognize it. The DJI programmers have done the same thing that Android does, which is to not recognize its own native file systems at the user interface level. They don't want to introduce incompatible file systems from Linux into a mostly Windows and Mac world, hence they stick with the universal formats of FAT and exFAT. They want to make life simple and reduce customer complaints.

Unfortunately, neither FAT nor exFAT are fault tolerant file systems, but DJI forces us to use those formats. They are NOT journaling file systems and can therefore corrupt easily during a power bump or an unclean dismount, which does happen. Either though Windows is supposed to allow you to just "yank out" removable media, you are best to always eject it cleanly (dismount), in both Windows and Mac. If for any reason you get a message you can't, it's busy and yanking the card while it is in use is almost a surefire way to cause an error that requires fixing or maybe even reformatting.

Whether a format erases all data can be answered with the word, depends. A quick format in Windows does not erase data, but only resets a few housekeeping files, in simple terms. A regular format in Windows overwrites data during formatting since Windows 7 came out. Mac does not overwrite data during formatting. It is similar to a Windows quick format. Overwriting data, i.e. wiping, takes a lot of time, especially if it is a platter drive, but we don't have such in drones, only flash media, which is an entirely different beast altogether.

The memory cells in flash media have a finite number of times of reads and writes before they fail. It would not do if different portions of the card failed soon and other much later in life, so the manufacturers created proprietary wear-leveling algorithms that move the data around so that it wears out evenly. These algorithms are also 'aware' when files are deleted so they can more efficiently make use of space during wear-leveling. Thus deleting files normally doesn't delete the data, but the wear leveling algorithm kicks in and therefore deleted data gets overwritten fairly quickly. Just how quickly? Depends on lots of factors well beyond the scope of this conversation.

So basically the more a flash media card is used (reads / writes), the more quickly it will eventually fail. That said, doing a lot of formatting adds to the number of reads/writes and, in theory, wears the card out sooner. In practice, I have never worn out a flash memory card yet. I've lost them, drowned them, broken them, had them stolen, but more often than not I retire them and replace them with larger and faster media.

There are times when despite all the care, the card just won't read, usually on the computer. On the computer, I usually can remedy that by plugging into a different port or by rebooting. If that fails, I resort to a disk repair and as a last resort, I reformat. If I have to reformat, I try to rescue any data first. I find the Access Data's FTK imager will allow me to get the data when the computer can no longer mount the file system. It's free and if it can read it physically, you can copy your data off. Just search and download. There's no substitute for frequent data backups. Usually when you lose data, you haven't been diligent in that regard.

Just a few tips that will help you:
1) Format your media in the device for which it was intended, as it will choose the best format and underlying settings (not all formatting is equal)
2) Use good, up-do-date, card readers. Take care with cables and have spares. Swap out a reader or cable if you are having issues. Trying a different port or reboot.
3) ALWAYS do a clean dismount. Eject the media properly in either Windows or Mac. Failing to do so causes more issues than any other reason in my experience. Many people see me doing this in Windows and ask what I'm doing and why. Clearly they know little about properly dismounting media devices. Mac users seem to understand "dismounting" as they get scolded by the operating system if they fail to do so.
4) Don't let other people use your media cards in their devices. They may have malware or they may not care to properly eject the media card.
5) Always backup your data. Minimize how much you can lose at any given point in time.

Best and enjoy flying...

Steve

Thanks Steve for this detailed and very informative post. I have always done a clean or "safe" eject from my computers. Occasionally when I put an SD Card from one of my drones into my computers, Windows will tell me it needs to be fixed. It always seems to fix it just fine but often wondered if I should re-format it later when I put it back into my drone. Your thoughts?

 

[sbunting]

Thanks Steve for this detailed and very informative post. I have always done a clean or "safe" eject from my computers. Occasionally when I put an SD Card from one of my drones into my computers, Windows will tell me it needs to be fixed. It always seems to fix it just fine but often wondered if I should re-format it later when I put it back into my drone. Your thoughts?

As long as you have your data safe, I'd format often and let DJI do it on board. It starts you off with a known clean slate. In my business, it's a best practice to use "sterile" media, so I like to start clean. That way you start your flight with maximum good disk space with the least likelyhood of disk errors. Like I've said, I've never in my life worn out an SD card ;->

 

[Eagle Eye 62]

As long as you have your data safe, I'd format often and let DJI do it on board. It starts you off with a known clean slate. In my business, it's a best practice to use "sterile" media, so I like to start clean. That way you start your flight with maximum good disk space with the least likelyhood of disk errors. Like I've said, I've never in my life worn out an SD card ;->

Awesome, thanks!

 

[Yorkshire_Pud]

If you want a bucket full of misinformation, start a discussion about SD cards and formatting. Generally, whether a MicroSD card is formatted on the device in which it is intended to function or on a computer shouldn't matter, but sometimes it does.

Assuming everything you say is correct ...GREAT POST.

Ummm minor point, in the 7th paragraph you refer to "regular format", at a guess most folks here or drone users in general, would be primarily windows users like myself and regard the quick format as the norm/regular method of formatting. Your use of "regular format" might be confusing to them.
As you say the alternative to the quick format ("low level"?) does take a long time and hence is probably not popular amongst most windows users.

With regards to 'formatting on the intended device', from posts that I have read it seems that formatting the card in a DJI drone resets the file number counter for video and photograph names and that this is a problem for some people.
I can not say from first hand experience if this is correct because, in general, I do not reformat a card that has been emptied and if I do I tend to use windows. But it may be something to bear in mind.
Yes I know there are simple ways of 'correcting' the problem but some people throw a hissy fit when that is suggested and rant that "DJI should fix this problem".

Question, can you explain, in every day language, not computer speak, journalised file systems please? I have seen mention of them when setting up a dual boot machine but haven't looked into them.
Question, does "Access Data's FTK imager" recover ALL types of data? I think some 'recoverers' are limited to photos and videos.

Might I suggest paragraphs 8 & 9 are changed to a different font colour etc. so as to stand out i.e. I think they should be highlighted.

 

[sbunting]

Assuming everything you say is correct ...GREAT POST.

Ummm minor point, in the 7th paragraph you refer to "regular format", at a guess most folks here or drone users in general, would be primarily windows users like myself and regard the quick format as the norm/regular method of formatting. Your use of "regular format" might be confusing to them.
As you say the alternative to the quick format ("low level"?) does take a long time and hence is probably not popular amongst most windows users.

With regards to 'formatting on the intended device', from posts that I have read it seems that formatting the card in a DJI drone resets the file number counter for video and photograph names and that this is a problem for some people.
I can not say from first hand experience if this is correct because, in general, I do not reformat a card that has been emptied and if I do I tend to use windows. But it may be something to bear in mind.
Yes I know there are simple ways of 'correcting' the problem but some people throw a hissy fit when that is suggested and rant that "DJI should fix this problem".

Question, can you explain, in every day language, not computer speak, journalised file systems please? I have seen mention of them when setting up a dual boot machine but haven't looked into them.
Question, does "Access Data's FTK imager" recover ALL types of data? I think some 'recoverers' are limited to photos and videos.

Might I suggest paragraphs 8 & 9 are changed to a different font colour etc. so as to stand out i.e. I think they should be highlighted.

I clarified the issue about a quick format vs a 'regular format' with an edit to the above. A regular format, one with the Quick Format box unchecked, isn't a low level format, but a format in which all the data is overwritten with zeros or random data, etc. A quick format does not overwrite data, just resets some of the 'front end' file system management features.

I'm not sure if the reset counter is impacted by formatting. Perhaps someone else can address that. I move cards around so much it's a non-issue with me.

Access Data's FTK Imager isn't so much a data recovery tool in the traditional sense, but it can be used as such. It can see drives at a physical level that won't mount in Windoze. That includes the Mac formats as well, which comes in handy. That is can display file systems that Windows can't or won't display, turns it into a recovery tool somewhat. That it allows all folders and files to be copied out, yes, it is a recovery tool. That it displays deleted files and allows them to be copied out, yes, it is a recovery tool It does not carve out data as a means of data recovery, but it can create an image that can be used for data recovery.

As for a journaling file system, Windows has NTFS, Linux has Reiser, ext2, 3 & 4, and Apple has HFS+ and APFS. There are others. With this type of file system data is first written to a journal and a dirty bit is set. Once the data is written to the actual final storage location (cluster, etc) then the dirty bit it cleared. If the process were halted before the dirty bit were cleared, upon restart, the dirty bit would tell the file system to restore it to stable state, using data stored in the journal, and data loss would be prevented. I don't store any data I care about on FAT or exFAT file system, only journaled ones. That's it in a nutshell.

Paragraphs 8 & 9 are in blue, as you suggest...

Enjoy...

Steve

 

[Meta4]

I'm not sure if the reset counter is impacted by formatting. Perhaps someone else can address that.

For drones using DJI Fly, it is.
With DJI Go 4 it didn't matter, the app settings allowed the choice of Continuous File Index - file numbering continues across mulitiple cards .. or
Reset, which resets the file numbering to 001 for each new card

But with DJI Go there is no setting in the app and if you format in the drone, the numbering restarts at 001 for each card.
But if you format in the computer, each new card continues the file numbering across new cards.

It's bad enough that DJI file numbering only goes to 999 before restarting at 001.
But having each card restart at 001 is a serious and unnecessary inconvenience for flyers doing serious photography or mapping where they generate lots of images in a day.

I generate a lot of images and for 8 years flying DJI, I've always formatted each card in the computer after copying the files across.
I've never had any problem caused by that.

 

[sbunting]

For drones using DJI Fly, it is.
With DJI Go 4 it didn't matter, the app settings allowed the choice of Continuous File Index - file numbering continues across mulitiple cards .. or
Reset, which resets the file numbering to 001 for each new card

But with DJI Go there is no setting in the app and if you format in the drone, the numbering restarts at 001 for each card.
But if you format in the computer, each new card continues the file numbering across new cards.

It's bad enough that DJI file numbering only goes to 999 before restarting at 001.
But having each card restart at 001 is a serious and unnecessary inconvenience for flyers doing serious photography or mapping where they generate lots of images in a day.

I generate a lot of images and for 8 years flying DJI, I've always formatted each card in the computer after copying the files across.
I've never had any problem caused by that.

"I generate a lot of images and for 8 years flying DJI, I've always formatted each card in the computer after copying the files across.
I've never had any problem caused by that." As long as that workflow gets your work done and you have no problems, why change it? My recommendations about formatting with the device in which the card is intended to be used by those folks who may not be familiar with formatting options, in which case it's better to let DJI do it for you. Keep it simple as possible. I think most problems are caused by pulling out cables or cards without dismounting them. If you do so while the operating system is indexing or has files open for any reason, you've screwed the proverbial pooch. Unfortunately, the OS doesn't tell you when this is happening unless you start to dismount and it tells you it is busy. I'd be willing to bet that you are one who ejects / dismounts properly since you've had no issues ;->

As to the numbering issue, DJI Fly seems an immature product in many regards. They failed to put many of the good features in Go 4 into it, numbering being but one. I use Exiftool for organizing my files. As soon as they come off the card, the originals go to a repository where exiftool organizes them in folders according to the date they were created, based on the date in the exif data itself. They can also be renamed by the same tool and the renaming can reflect the time taken to the second, with a switch to append a 1, 2, 3 etc in the event of a duplicate name. The tool can be used on Windoze or Mac. It runs by command line, although there is likely a GUI out there that runs on top of it. Below is command to organize by directory and the result. For this reason, DJI's lack of numbering doesn't bother me in the least.

 

[Yorkshire_Pud]

DJI Fly seems an immature product in many regards.

I think that's being VERY, VERY kind

 

[sbunting]

I think that's being VERY, VERY kind

I've been accused of being very very honest to a fault
So this is my attempt at being 'kind', which equates to being less than honest, as you have so noted
OK - DJI Fly is horrible. It reminds me of a basic programmer's first project in Programming 101

 

[FlyDork]

I never had a problem until today. I just reformatted the card (FAT32) in my MacBook and it worked fine after in my Mavic Air/Macbook after that.