How to extract geotagged frames from video for use in photogrammetry

[ZeeDoktor]

Hi all,

This might be useful to some of you wanting to do photogrammetry from videos you've taken with your drones and prepare them for processing with a service like WebODM. This creates excellent orthophotos, point clouds, and 3D models from your scenes.

This is not point and click, this is all command line stuff. But it's free. And if you use this often and with varying requirements, you'll soon discover that being able to script all of this is far superior to having to point and click yourself through windows...

Speaking of windows... I don't use (capital) Windows so I can't help you figuring out how to run it there, but I am told it's easy and all the software packages you need are available.

First you need to install some freely available software:
- ffmpeg FFmpeg
- python Python Releases for Windows
- exiftool ExifTool by Phil Harvey
- my scripts srt_tag.py and srt_concat.py tagtools.zip

Remember these are command line tools, you can generally run them with the -h command for help. For example, run my scripts using the command line

python srt_tag.py -h

or

python srt_concat.py -h

to figure out what parameters you need to give it.

If all of your video frames are split over multiple videos, skip this and read further down how to concatenate the videos and SRT files first.

If using a single video, process as follows:

1. convert video footage to frames:

ffmpeg -i DJI_0123.MP4 -vf fps=1 framesdirectory/%04d.jpg

fps=1 means extract one frame for every second of movie. If you want one frame every 2 seconds, use fps=0.5. framesdirectory is simply the name of the directory where you want the frames to go.

2. Then you need to geotag the extracted frames. Video frames do not have that built in automatically unlike still photos. But this can to a large degree be done using the SRT file:

python srt_tag.py -s DJI_0123.SRT -d framesdirectory/ -p 30 -x jpg -f 1

-p 30 means the original footage was shot at 30 frames per second, -x is the file extension, you can also export png if you want in the ffmpeg export step above (creates much improved quality orthophotos, but the png's are 10x the size of the jpg). -f 1 means the frames are at 1 frame per second in the original footage. This is important so it knows which SRT frame number to use for the data.

3. Then you simply add those frames to your WebODM project and hit upload and go. Done!

If you need concatenate the videos first, follow these instructions:

1. Concatenate the videos. Create a file (I called that files.txt) that has a line like this for every video file you want to add to the resulting video:

file '/Volumes/Video/Footage/2022-12-16/Hill/DJI_0251.MP4'
file '/Volumes/Video/Footage/2022-12-16/Hill/DJI_0252.MP4'

2. Create the output video from this using this command:

ffmpeg -f concat -safe 0 -i files.txt -c copy output.mp4

3. Create a concatenated SRT file that has the correct data for each frame:

python srt_concat.py -i concat_files.txt -o output.srt

4. Now export the frames, same as above:

ffmpeg -i output.mp4 -vf fps=1 framesdirectory/%04d.jpg

5. And geotag the output files, again, same as above:

python srt_tag.py -s output.srt -d framesdirectory/ -p 30 -x jpg -f 1

6. Upload to your WebODM project and done!

Hope this helps!

 

[Newlands]

Dr Zee,

This is fantastic! Thanks for the well-documented workflow.
On Windows 10, I had to copy exiftool(-k).exe into the TagTools folder with your scripts and renamed it "exiftool.exe".

Also, it's running a little slow and spitting out a disconcerting error on every frame:

"Warning: Invalid date/time (use YYYY:mm:dd HH:MM:SS[.ss][+/-HH:MM|Z]) in ExifIFDateTimeOriginal (PrintConvInv)
Error: File not found - 10:55:56.259000'
Error: File not found - shutter:
Error: File not found - 1/160.0'
1 image files updated
3 files weren't updated due to errors"

I'm not sure why it doesn't like the timestamp... The .SRT from this drone also has a few additional fields (zoom, pitch, yaw and roll) but I can't see how this would upset the parser:

-----------
1
00:00:00,000 --> 00:00:00,033
<font size="28">FrameCnt: 1, DiffTime: 33ms
2023-04-15 10:55:41.263
[iso: 100] [shutter: 1/160.0] [fnum: 11.0] [ev: 0] [color_md : default] [ae_meter_md: 1] [focal_len: 24.00] [dzoom_ratio: 1.00], [latitude: 34.081893] [longitude: -118.227260] [rel_alt: 109.241 abs_alt: 175.322] [gb_yaw: -173.6 gb_pitch: -7.6 gb_roll: 0.0] </font>
-------------

Despite the error messages, the app seems to be doing it's job -- Thanks again!!!

-Donald Newlands

 

[ZeeDoktor]

Hi Donald,

thanks for the kind feedback. I haven't actually used the code since I posted... glad someone else finds my effort useful, it was just a one-off for me!

srt_tag.py parses the properties by name, so having more properties shouldn't upset things as you state.

I'm not using windows often, but from memory there are some command line oddities, it may not be accepting of single quotes vs double quotes. Perhaps try replacing

-DateTimeOriginal='%s'

with

-DateTimeOriginal=\"%s\"

on line 79 of srt_tag.py.

Another issue might be the language locale, although in this case I don't think that's the issue. But still, for any others running into problems with the parsing:

srt_tag.py on line 44 parses the time from the frames as follows:

ts = datetime.strptime(line, "%Y-%m-%d %H:%M:%S.%f")

If your drone language locale uses a comma as the decimal point (e.g. German, Spanish, possibly others), simply replace that line with this:

ts = datetime.strptime(line, "%Y-%m-%d %H:%M:%S,%f")

and your parsing errors should go away.

Hope this helps!

 

[Newlands]

Zee, thanks for your reply! Yes, it seems to be a Windows string thing... Changing -DateTimeOriginal='%s' to -DateTimeOriginal=\"%s\" fixed the date error and changing -ShutterSpeedValue='%s' to -ShutterSpeedValue=\"%s\" fixed two more.

Another more problematic issue: the latitude and longitude are inverted in the exif. Our video, shot near Los Angeles, CA is showing up somewhere SE of Perth, Australia... I tried inverting the longitude and latitude numbers (lat=lat * -1) but they still come out in Australia. I'm looking into the exiftool documentation to see if there's anything I can do to change how decimal lat/lon is interpreted as degrees/min/sec.

 

[Newlands]

Okay, I solved the latitude/longitude inversion issue: the exiftool has parameters that set which part of the globe you're refering to:

-GPSLatitudeRef=N -GPSLongitudeRef=W works for me in the US but you might need
-GPSLatitudeRef=S -GPSLongitudeRef=E if you live in Australia as Zee does

I'm surprised that the positive or negative sign in the decimal arc degrees format is ignored by exiftools. It's also not helpful that Windows does not display N,S,E,W with GPS coordinates in degrees, minutes, seconds in file details...

I have posted revised srt_tag_02.py code -- tested for Windows in North America.

 

[ZeeDoktor]

Thanks for the fixes, I've modified your changes further to automatically determine N/S/E/W references.

 

[Miro_Rava]

This reply is copied from another similar thread where i wrote the same thing:

Hi guys, I just encountered this problem and I couldn't find any ready to use tool for this, so I created my own with a gui in python, i also packaged it in an exe file, you can do this by yourself using auto-py-to-exe.
It basically takes as input the folder containing the videos and maybe also photos, the starting altitude (since as geotag dji uses relative altitude and not absolute), and the interval in seconds at which each frame has to be saved.
Videos without .SRT files will be discarded automatically.
Using this, I basically dump all my videos and photos of a site inside a folder, and then run the tool.
After I can use the folder with reality capture or other photogrammetry softwares.

I think this is a cool tool since it is self contained without the need for exif tools or others. Obviously to run it you need python and some of the libraries installed, but you can also use the .exe directly which is truly self contained without the need for python
for the ones that want a simple solution without installing python, i have the exe hosted on my website at this link:
https://www.miro-rava.com/documents/DJI_GPS_Video_Editor.exe
I didn't put it on github because it is not a finished project yet, this exe opens a gui that does what i explained above

----this is the gui with customtkinter: (save it in a file and name the extension as .pyw

import customtkinter
import subprocess
import sys
import os

customtkinter.set_appearance_mode("light")  # Modes: "System" (standard), "Dark", "Light"
customtkinter.set_default_color_theme("green")  # Themes: "blue" (standard), "green", "dark-blue"

app = customtkinter.CTk()
app.geometry("600x280")
app.title("Altitude Image Processor")


def resource_path(relative_path):
    """ Get absolute path to resource, works for dev and for PyInstaller """
    try:
        # PyInstaller creates a temp folder and stores path in _MEIPASS
        base_path = sys._MEIPASS
    except Exception:
        base_path = os.path.abspath(".")

    return os.path.join(base_path, relative_path)


def start_batch():

    subprocess.Popen(["python", resource_path("alt_changer.py"), folder_path.get(), entry_1.get(),entry_2.get()])

def browse_folder():
    # Allow user to select a directory and store it in global var
    # called folder_path
    global folder_path
    filename = customtkinter.filedialog.askdirectory()
    folder_path.set(filename)


frame_1 = customtkinter.CTkFrame(master=app)
frame_1.pack(pady=20, padx=20, fill="both", expand=True)

folder_path = customtkinter.StringVar()
folder_path.set("No Folder Selected")

button_1 = customtkinter.CTkButton(text="Chose Image Folder", master=frame_1, command=browse_folder)
button_1.pack(pady=10, padx=10)

label_2 = customtkinter.CTkLabel(master=frame_1, textvariable=folder_path, justify=customtkinter.LEFT)
label_2.pack(pady=10, padx=10)

entry_1 = customtkinter.CTkEntry(master=frame_1, placeholder_text="Enter Altitude in m")
entry_1.pack(pady=10, padx=10)

entry_2 = customtkinter.CTkEntry(master=frame_1, placeholder_text="Frames Interval in s")
entry_2.pack(pady=10, padx=10)

button_2 = customtkinter.CTkButton(text="Start Batch Processing", master=frame_1, command=start_batch)
button_2.pack(pady=10, padx=10)

if __name__ == '__main__':
    app.mainloop()

-----and this is the main file that processes the videos: (save it in the same folder as the gui in a .py file called "alt_changer.py"

import os
import sys
import cv2
import piexif
import re
import exif
from PIL import Image
from fractions import Fraction

files = os.listdir(sys.argv[1])
percentage = 0

def progress_bar(current, total, bar_length=20):
    fraction = current / total

    arrow = int(fraction * bar_length - 1) * '-' + '>'
    padding = int(bar_length - len(arrow)) * ' '

    ending = '\n' if current == total else '\r'

    print(f'Progress: [{arrow}{padding}] {int(fraction*100)}%', end=ending)

def parse_srt_file(srt_path):
    latitudes = []
    longitudes = []
    altitudes = []

    with open(srt_path, 'r') as file:
        srt_content = file.read()

    pattern = r'\[latitude: ([\d.-]+)\] \[longitude: ([\d.-]+)\] \[altitude: ([\d.-]+)\]'
    matches = re.findall(pattern, srt_content)
    for match in matches:
        latitude = float(match[0])
        longitude = float(match[1])
        altitude = float(match[2])

        altitudes.append(altitude)
        latitudes.append(latitude)
        longitudes.append(longitude)

    return latitudes, longitudes, altitudes

# Convert the latitude and longitude to the required format (Rational)
def degrees_to_rational(number):
    degrees = int(abs(number))
    minutes = int((abs(number) - degrees) * 60)
    seconds = int(((abs(number) - degrees - minutes / 60) * 3600) * 100)

    return [(degrees, 1), (minutes, 1), (seconds, 100)]

print("Elaborating Videos with Timestamps if present:")

for videoPath in files:
    if videoPath[-4:] in [".MOV", ".mov", ".MP4", ".mp4"]:
        video_path = os.path.join(sys.argv[1], videoPath)
        video_name = os.path.splitext(os.path.basename(video_path))[0]
        srt_path = os.path.join(sys.argv[1], video_name + ".srt")
        capture = cv2.VideoCapture(video_path)
        frame_count = int(capture.get(cv2.CAP_PROP_FRAME_COUNT))
        frames_to_extract = range(0, frame_count-1, int(float(sys.argv[3]) * capture.get(cv2.CAP_PROP_FPS)))
        try:
            latitudes, longitudes, altitudes = parse_srt_file(srt_path)
        except FileNotFoundError:
            print(f"WARNING ---> Skipping Video: {video_name} ---> NO .SRT File found")
            continue
        for frame_index in frames_to_extract:
            capture.set(cv2.CAP_PROP_POS_FRAMES, frame_index)
            success, frame = capture.read()

            if success:
                frame_path = os.path.normpath(os.path.join(sys.argv[1], f'{video_name}_frame_{frame_index}.jpg'))
                cv2.imwrite(frame_path, frame)
                try:
                    frame_latitude = latitudes[frame_index]
                    frame_longitude = longitudes[frame_index]
                    frame_altitude = altitudes[frame_index]+int(sys.argv[2])
                except Exception as e:
                    print(f"Image not processed because ---> {e}")
                    continue

                if frame_latitude is not None and frame_longitude is not None and frame_altitude is not None:
                    try:
                        # Open the image file
                        image = Image.open(frame_path)

                        # Get the existing EXIF data
                        exif_dict = piexif.load(frame_path)

                        # Preserve existing GPS data
                        gps_info = exif_dict.get("GPS", {})
                        altitude_ref = gps_info.get(piexif.GPSIFD.GPSAltitudeRef, 0)
                        gps_version = exif_dict.get(piexif.GPSIFD.GPSVersionID, (2, 3, 0, 0))

                        new_lat_rational = degrees_to_rational(frame_latitude)
                        new_lon_rational = degrees_to_rational(frame_longitude)

                        # Update the GPS data in the EXIF metadata
                        exif_dict["GPS"] = {
                            piexif.GPSIFD.GPSLatitudeRef: 'N' if frame_latitude >= 0 else 'S',
                            piexif.GPSIFD.GPSLatitude: new_lat_rational,
                            piexif.GPSIFD.GPSLongitudeRef: 'E' if frame_longitude >= 0 else 'W',
                            piexif.GPSIFD.GPSLongitude: new_lon_rational,
                            piexif.GPSIFD.GPSVersionID: (2, 3, 0, 0),
                            piexif.GPSIFD.GPSAltitude: Fraction.from_float(frame_altitude).limit_denominator().as_integer_ratio(),
                            piexif.GPSIFD.GPSAltitudeRef: 0,
                        }

                        # Encode the EXIF data and save it back to the image
                        exif_bytes = piexif.dump(exif_dict)
                        image.save(frame_path, exif=exif_bytes)

                        print(f'{video_name}_frame_{frame_index}.jpg')
                    except Exception as e:
                        print(f"Image not processed because ---> {e}")


                else:
                    print(f"No GPS data found for frame: {frame_path}")
            else:
                print(f"Error extracting frame at index {frame_index}")

        capture.release()

print("Elaborating Images:")

for imagePath in files:
    if imagePath[-4:] in [".JPG", ".PNG", ".jpg", ".png"]:
        percentage += 100/len(files)
        full_imagePath = os.path.join(sys.argv[1], imagePath)
        with open(full_imagePath, 'rb') as image_file:
            img = exif.Image(image_file)
            img.gps_altitude = img.gps_altitude + int(sys.argv[2])
        ext = full_imagePath[-4:]
        tempPath = full_imagePath[:-4]
        with open(f"{tempPath}_mod{ext}", 'wb') as test_image_file:
            test_image_file.write(img.get_file())
        os.remove(full_imagePath)
        print(f"{percentage:.2f}%  --> Changed GPS Altitude of: {full_imagePath[-12:-4]}")
print("100.00% --> Done!!")

 

[noelnac]

Thanks for the fixes, I've modified your changes further to automatically determine N/S/E/W references.

just replace latRef with lonRef in lines 84 & 86 and all will working fine.
if lat < 0:
latRef = "S"
else:
latRef = "N"
if lon < 0:
lonRef = "W"
else:
lonRef = "E"

 

[marco-re1234]

Hi, thank you for share your useful tool!
I tried to model a building with webodm using photos extracted with your script but the model is upside down and if i check the altitude values in the frames, they are all inverted. The altitude seems to increase in the lower ones. How to fix that?

 

[Miro_Rava]

Hi, thank you for share your useful tool!
I tried to model a building with webodm using photos extracted with your script but the model is upside down and if i check the altitude values in the frames, they are all inverted. The altitude seems to increase in the lower ones. How to fix that?

are you referring to my tool?
If Yes, make sure that in the field of altitude you insert the altitude in meters of the starting point of your drone.

If you have the same problem you can send me a short clip and respective srt file in order to test if with your video is working correctly, I just checked the code but I didn't find any errors and also just yesterday I used the tool to elaborate data of a building and it was correctly elaborated.

another thing you can try is to take two frames that you generated (one that you are sure it was taken higher than the other) and put them in an online exif viewer to check that the two altitude are correctly one higher than the other, at that point the problem is probably WebOdm. If you never tried , try maybe RealityCapture, it is fast, accurate and hyperdetailed, you can use the PPI (Pay Per Input) license, yesterday I payed 6 usd for 2000 12mpx images, it is really cheap.

let me know if it works

 

[marco-re1234]

are you referring to my tool?
If Yes, make sure that in the field of altitude you insert the altitude in meters of the starting point of your drone.

If you have the same problem you can send me a short clip and respective srt file in order to test if with your video is working correctly, I just checked the code but I didn't find any errors and also just yesterday I used the tool to elaborate data of a building and it was correctly elaborated.

another thing you can try is to take two frames that you generated (one that you are sure it was taken higher than the other) and put them in an online exif viewer to check that the two altitude are correctly one higher than the other, at that point the problem is probably WebOdm. If you never tried , try maybe RealityCapture, it is fast, accurate and hyperdetailed, you can use the PPI (Pay Per Input) license, yesterday I payed 6 usd for 2000 12mpx images, it is really cheap.

let me know if it works

I was referring to the ZeeDoktor scripts, the subject of this thread. I tried your editor too but with my video it doesn’t work. I have a Mini 3 Pro and maybe, as I have read in your dedicated thread, it's a format problem. I'm just an hobbist I don't want to spend money on it.

 

[Miro_Rava]

I was referring to the ZeeDoktor scripts, the subject of this thread. I tried your editor too but with my video it doesn’t work. I have a Mini 3 Pro and maybe, as I have read in your dedicated thread, it's a format problem. I'm just an hobbist I don't want to spend money on it.

Okay, no problem! If you want to help me and share with me a short recording from your mini 3 pro and the corresponding subtitle, i can make my tool work too, I'm slowly adding support for all different types of dji drones, but i need example srt files and video to do that

 

[ZeeDoktor]

Hi @marco-re1234, sorry for chiming in late. Are you still having this problem?

 

[zmlski]

I completed Step 1 without issue.

I tried Step 2 and got an error:
PS C:\Users\zacha> python "C:\Users\zacha\Downloads\srt_tag.py" -s "C:\Users\zacha\Desktop\DCIM\100MEDIA\DJI_0003.SRT" -d "C:\Users\zacha\Desktop\Test" -p 60 -x png -f .5 -e "C:\Users\zacha\Desktop\exiftool.exe"
Traceback (most recent call last):
File "C:\Users\zacha\Downloads\srt_tag.py", line 89, in <module>
(args.exiftoolpath, args.dir, frame_file, args.file_extension, lat, lon, alt, latRef, lonRef, ts.strftime("%Y-%m-%d %H:%M:%S.%f"), iso, focal_len, shutter, fnum)
^^^
NameError: name 'alt' is not defined. Did you mean: 'all'?

Any idea what is going on?

 

[ZeeDoktor]

That means it couldn't find the "rel_alt" key in the metadata of the video. insert

print(line)

after line 50 and post one line here.

 

[wulfric777]

Traceback (most recent call last):
File "D:\Belajar Python\tagtools\srt_tag.py", line 34, in <module>
frame_number = int(line)
^^^^^^^^^
ValueError: invalid literal for int() with base 10: '\n'

any solution?

 

[Miro_Rava]

Traceback (most recent call last):
File "D:\Belajar Python\tagtools\srt_tag.py", line 34, in <module>
frame_number = int(line)
^^^^^^^^^
ValueError: invalid literal for int() with base 10: '\n'

any solution?

It is taking an into of a brake line (\n)
This depends on the format of your srt

If you don't want to spend time with python, you can try my tool downloadable here:

https://miro-rava.com/documents/DJI_Image_Processor.exe

It has a tool for converting flir images into tiffs and also a tool for extracting geotagged frames from a dji video.
It should support most of the srt from dji

Let me know if it helps!