Motivation

Drone Mapping Data - Where it Starts

Volume and Variety - Double Trouble

uasimg Overview

R package with functions to help you:

• create catalogs of your images
• record your flight metadata
• create GIS data of your data collections
• store your images in a data structure that will allow you to find them again
• share your data / metadata with others
• utilities for unstitchable images
• deal with annoying data management tasks

Data Catalog Tour

Origins

https://is.gd/uas_data_aag2019

Why R?

• full featured scripting language
  
• wrappers for open source libraries (exiftool, gdal, geos)
  
• good reporting engine
  
• good deployment system
  
• good tools to create GUIs

Step 1. Extract Image EXIF Data

ptpinole_info = uas_info("d:/uas/ptpinole/west/march2020/dcim")

This will:

1. extract the image coordinates
2. read the sensor type, location, yaw, focal length, etc.
3. compute image footprints
4. create an minimum convex polygon (MCP) for the entire flight
5. read flight metadata
6. cache all the results
7. return a “Flight Info” object

You can process multiple directories at once

A “Flight Info” object by itself is not very useful!

Step 2. Create / Edit Flight Metadata

Image metadata vs Flight Metadata

Entering Flight Metadata

Recommended way to record flight metadata - Notepad!

• Any field name allowed
  
• Defaults:

uas_flds_oem()

##  [1] "name_short"  "name_long"   "description" "proj"        "loc"        
##  [6] "pilot"       "contact"     "uav"         "data_url"    "tags"       
## [11] "notes"

Flight Metadata Helper Function

You don’t have to start from scratch!

uas_metadata_make(ptpinole_info)

• create a metadata.txt file for every image directory in x
  
• you can customize field names with uas_setflds()
  
• can pre-populate values from a template (example)
  
• open them in text editor

Re-run uas_info() after you edit flight metadata!

Step 3. Export flight geometry as GIS files

If desired, you can export the flight geometry:

uas_exp_kml(ptpinole_infox)
uas_exp_shp(ptpinole_info)

• image center points
• footprints
  
• flight MCP
  
• default ouptut folder is a ‘map’ subfolder of the image directory

Move Your Data into Standard Directory Trees

You can move your data by hand or with a function:

uas_move(ptpinole_info)

Function options:

• move or copy images
• create the directory tree from a template text file:

• placeholders in {} will be replaced with flight and/or image metadata
  
• you can use any flight metadata field and a subset of image metadata fields
  
• the first directory should be where you want the images from a single flight to go

Create a Flight Summary Report

Create HTML flight summary for one or more Flight Info object(s):

uas_report(ptpinole_info)

Report options:

• report title
• HTML header / footer
• group image locations into ‘clusters’
• show image thumnbnails
• custom header / footer (in HTML)
• show the directory where the images are saved on your local machine
• hyperlinks to download the KML files
• download satellite image thumbnail (requires Google API key)
  
• output directory (default is ‘map’ sub-directory)

Group Flight Summary Reports into a Table of Contents

Feed a list of Flight Summary HTML files.

uas_toc(c("d:/uas/ptpinole/west/march2020/dcim/map/ptpinole_mar2020.html",
          "d:/uas/ptpinole/west/jul2020/dcim/map/ptpinole_jul2020.html",
          "d:/uas/ptpinole/west/oct2020/dcim/map/ptpinole_oct2020.html"))

It will extract flight metadata fields from the HTML file for the table-of-contents.

Function options:

• title & HTML header/footer
  
• copy all the individual Flight Summary HTML files to one place
  
• show a map of all the flight areas
  
• include hyperlinks to download all flight areas as a single KML

Sample code that does everything

library(uasimg)

## Set the working directory to the parent directory of the image folders:
setwd("D:\\Pix4D\\HREC\\Watershed1\\Data\\2017-01-16_X5")

hrec_dirs <- c("Flight01_1514_1526_400ft", "Flight02_1532_1540_400ft")
file.exists(hrec_dirs)

## Make metadata text files
# uas_metadata_make(hrec_dirs, make_new = TRUE, overwrite = FALSE, open = TRUE, use_system_editor = TRUE)

## Extract flight metadata
hrec_info <- uas_info(hrec_dirs, fp = TRUE)

## Create reports
hrec_rpt <- uas_report(hrec_info, thumbnails = FALSE,
                       attachments = c("mcp_kml", "ctr_kml"),
                       footer_html = "D:\\GitHub\\uasimg\\headers_footers\\footer_igis.html",
                       png_map = TRUE, group_img = FALSE, show_local_dir = TRUE,
                       overwrite_html = TRUE, open_report = TRUE)

## Create a TOC
hrec_toc <- uas_toc(hrec_rpt,
                    toc_title = "HREC Watershed2, January 2017",
                    output_dir = ".",
                    fltmap_base = list(list(kml_fn = "D:\\Pix4D\\HREC\\Baselayers\\hopland.kml",
                                            color = "red",
                                            weight = 2)),
                    fltmap_kml = TRUE,
                    footer_html = "D:\\GitHub\\uasimg\\headers_footers\\footer_igis.html",
                    open_toc = TRUE, overwrite_toc = TRUE)

Utilities for Unstitchable Images

World Files

‘World Files’ are plain-text files (also called sidecar files) with projection info for an image / dataset.

GIS software like ArcGIS Pro and QGIS use these files to display images in the proper place and size.

Generate with:

uas_worldfile(ptpinole_info)

Function options:

• World File format (aux.xml, wld, or prj)
• model yaw rotation

Output:

With World files images will just “pop” into place (approximately):

Crop Out Overlap

If your images can’t be stitched, life is not over!

This function will crop each image, keeping the center part (least distorted)

uas_cropctr(ptpinole_info)

Function options:

• specify the crop dimensions in the image horizontal and vertical axis in meters

Example Output:

Not great, but…

• If the ground is flat and the features are short, might be good enough.
  
• If you’re counting things in individual images, might be good enough.
  
• If you know you won’t be able to stitch images (i.e., open water), consider flying at “zero percent” overlap and use this technique to verify you’re not double-counting.

Annoying Tasks: Convert file formats

Convert between JPG, TIF, and DNG without losing all the image metadata (EXIF info):

uas_convert()

In the pipeline…

Scaling for Production

• uas_dirs_find() - find all directories with image files
  
• automate production with an Excel based control panel
  
• tips and tricks for production processing
  
• Shiny GUI (?)

Tools to get to “one flight per folder”

• multiple flights in one folder - split
  
• one flight spread across multiple directories - merge
  
• handle the details with renaming files, flight metatadata, log and calibration files, etc.

Make your data findable

• generate XML files for Google Dataset Search
  
• publishing your data catalogs as AGOL web maps

Catalog Processed Data

• add links to processed data
  
• ingest the Pix4D / Metashape quality reports (processing settings, quality metrics)

Summary

The need for data management tools and workflows increases as you:

• accumulate more and more data
  
• work in a team
  
• are mandated to archive your data for the long term
  
• are mandated to make the existence of your data known to others