Can a DAT file be split into smaller chunks? If so are any special characters needed at the end of one file and or at the beginning of the next file?

[Deleted member 103366]

I have just had a look at the csv of a DAT made from a fc_log.log file and it is enormous, 144.000+ lines.
It seems to contain records from at least 8 different days and what might be 9 flights but opening it with CsvView is ssslllloooooowwwwwww and saving the spreadsheet of the output csv takes a minute or so.
I am wondering if I can break it into more manageable chunks?

 

[eEridani]

Are you wanting to split a CSV or the .LOG file or the .DAT?

For the CSV, in Excel, you can jump down to a halfway point select the rest and cut and paste to a new worksheet. Then copy the header line. Doesn't help with the initial load tho.

In Wordpad, or Notepad++, you can do basically the same. Make two copies of the file, edit each copy to trim the top half and then the second copy trim the bottom half. Or, cut from one, create a new .csv and paste. Then also copy the header line across.

For the other file types, no clue.

 

[Deleted member 103366]

I want to split the log file itself.
Displaying the graphics produced by CsvView for the full log file is a bit much for my computer.
Even trying to open the log file in word etc. is taking a lllloooonnnnggggg time and that's just to see if I can see anything intelligible like dates.

 

[eEridani]

If the file is text based, with CRLF or similar line terminations, and you are running Windows, try Wordpad to open it (not Notepad); and if that takes a while try Notepad++ (a free tool that understands many programming languages and formats them properly). Notepad++ won't read the log, but it is efficient at handling large text files.

Notepad++

notepad-plus-plus.org

 

[Deleted member 103366]

Notepad ++ opens it, thanks, 5,841,000 lines eek.
I see a few patterns but I need to pough through it to see if I can relate anything to date or flight changes

 

[TS_Thomas02]

I actually was wondering about the opposite scenario...merging two records.

Not that I do this or really need this, more of a hypothetical question. My scenario....I started a flight on a Mini-1, in Mavic Maven...maybe it was Litchi, doesn't exactly matter. Then I get the idea, while it's hovering there, does the DJI program connect? Answer...yes it does. Then I ponder what is happening with the flight log.txt file on my iPad. Well the answer to that appears that whichever program is in the foreground commits to logging data while the other one suspends logging. One had a takeoff, one had a landing. So for this, it's one of the few flights that I didn't bother keeping on Airdata.com. But I did upload it and that's how I can tell what was in each.

It does raise a question about ability to merge, but because of the propensity of these programs to get large, and the questionable nature of the control transfer, and the overall questionability of if this is even useful, I never pursued it. But if we do find a "header" or "footer" in these files, I'll be curious to know about it.

 

[Deleted member 103366]

I am in the process of merging summaries of the .txt flight logs for Mavic Mini, Phantom 3, Mavic 2 Pro, Mini 2 and the FPV, This is working on the csv output of, for the first 3 ,TXTlogToCSVtool, and PhantomHelp & Airdata for the last two, it is probably simple programming for someone who knows what they are doing but tricky for this numpty.
The columns are shifted about within the csv's depending on the processor i.e.TXTlogToCSVtool, PhantomHelp or Airdata .....and or the drone. The actual header names can vary between the processors too, so you/I need to know what each column records etc. to compare/merge them,......which is not that easy at times for me.
I am using linux to produce the summaries and the merger.