Intro to Spatial Data
Analysis with R
SCGIS 2023
Annual Conference
Getting Data from APIs
Getting Data from APIs
Many functions that import data from local files can also be used to download data from online sources, provided the URL returns data in a standard format. Examples:
library(sf)
sf_nb <- sf::read_sf("https://data.sfgov.org/resource/xfcw-9evu.geojson")
plot(sf_nb %>% st_geometry(), main = "San Francisco Neighborhoods")
ca_breweries_df <- read.csv("https://raw.githubusercontent.com/ucanr-igis/rspatial_data/master/data/ca_breweries.csv")
head(ca_breweries_df)## Name Address City State Phone Type
## 1 10 Mile Brewing Company 1136 E Willow St Signal Hill CA (562) 612-1255 Brewery
## 2 101 North Brewing 1304 Scott St Suite D Petaluma CA (707) 778-8384 Brewery
## 3 14 Cannons 31125 Via Colinas Suite 907 Westlake Village CA (818) 652-6971 Brewery
## 4 21st Amendment Brewery - San Leandro 2010 Williams St San Leandro CA Brewery
## 5 2Kids Brewing Company 8680 Miralani Drive San Diego CA (858) 480-5437 Brewery
## 6 32 North Brewing Company 8655 Production Ave Suite A Sand Diego CA 619-363-2622 Brewery
If you need to download a file programmatically, use
download.file().
If you need to download a zip file, you can download it to a temp
file, unzip the contents where they should go (with
unzip()), then delete the temp file.
Many data providers have a ‘preview’ page for a dataset, and (hopefully) a Direct Link.
You want to use the Direct Link with functions that accept a URL.
To get the direct link for a file on Google Drive, visit:
https://buildbrothers.com/gdrive-generator/
raster::getData() can download the following datasets
directly into R:
To specify which area you want, you either pass a country abbreviation (alt or GADM) or a latitude-longitude coordinate (SRTM and worldclim).
There’s another getData() function in another package,
so always use the raster::getData() prefix.
To see the three-character ISO3 codes for each country, run
raster::getData('ISO3')
Downloads are cached by default. If you don’t want the temporary
files saved, pass download=FALSE.
Let’s download and plot the District boundaries and DEM for Zambia.
## Warning in raster::getData(name = "alt", country = "ZMB", mask = "TRUE"): getData will be removed in a future version of raster
## . Please use the geodata package instead## Warning in raster::getData(name = "GADM", country = "ZMB", level = 2): getData will be removed in a future version of raster
## . Please use the geodata package insteadclass(zmb_alt); class(zmb_districts)
plot(zmb_alt, main="Districts of Zambia")
plot(zmb_districts, col=NA, border="black", add=TRUE)## [1] "RasterLayer"
## attr(,"package")
## [1] "raster"
## [1] "SpatialPolygonsDataFrame"
## attr(,"package")
## [1] "sp"
Using the raster::getData() to plot the administrative
boundaries of a country of your choice. See what the different values of
the level argument return.
iNaturalist is a global community of naturalists that use the iNaturalist app to share observations and communicate. Over 73m observations to date!
You can access iNaturalist data via the rinat API client package.
## install.packages("rinat")
library(rinat)
library(sf)
# Define spatial boundaries
sample_bounds <- c(38.44047, -125, 40.86652, -121.837)
## Get observations (first 100)
deer_df <- get_inat_obs(query = "Mule Deer", bounds = sample_bounds)
## Plot results
plot(deer_df$longitude, deer_df$latitude)
caladaptR provides functions to directly import climate data from Cal-Adapt, a climate data portal for the western USA. The data come in as data frames or rasters.
To retrieve data, you first construct a “request object”, which is like an order form. This gets fed into a function that fetches data (more info).
library(caladaptr)
sac_tasmax_cap <- ca_loc_pt(coords = c(-121.4687, 38.5938)) %>% ## specify a location
ca_gcm(gcms[1:4]) %>% ## specify climate model(s)
ca_scenario(c("rcp45","rcp85")) %>% ## select emission scenarios(s)
ca_cvar(c("tasmax")) %>% ## select climate variables
ca_period("year") %>% ## select a temporal aggregation period
ca_years(start = 2040, end = 2070) ## select start and end dates
sac_tasmax_tbl <- ca_getvals_tbl(sac_tasmax_cap, quiet = TRUE) %>%
mutate(temp_f = units::set_units(val, degF))
library(ggplot2)
ggplot(data = sac_tasmax_tbl,
aes(x = as.Date(dt), y = as.numeric(temp_f))) +
geom_line(aes(color=gcm)) +
facet_grid(scenario ~ .) +
labs(title = "Annual Maximum Temperature for Sacramento", x = "year", y = "temp (F)")
FedData is an R package implementing functions to automate downloading geospatial data available from several federated data sources, including:
Feature classes on ArcGIS.com or ArcGIS Enterprise may be directly imported into R if the layer is public, or you have an ArcGIS license that gives you access.
| Configuration | R Download |
|---|---|
| public with JSON enabled |
import with esri2sf
|
| not public but accessible from AGOL or an ArcGIS portal using your ArcGIS account | import using the R-ArcGIS Bridge |
ArcGIS.com and ArcGIS Portal generally support server-side querying (both attribute and spatial). This means you can download just the features you need!
See also:
API keys are like passwords that provide access to services that in theory (depending on the service) you could be billed for.
However unlike passwords, API keys are often transmitted unencrypted from web pages or scripts like R or Python. Careless developers will even hardcode their API key in their HTML code or R script, making it very easy to discover.
There are two things you can do to protect your API key.
Some cloud platforms (e.g., Google) allow you to limit an API key to specific services (e.g., just downloading background tiles, or just geocoding).
Some services allow you to limit the the API key to specific application(s) (i.e., only calls from specific IP addresses or domain names will be processed).
Some services (e.g., ESRI Geocoding Service) allow you to specify an expiration date on API keys.
Although it’s very convenient to simply paste your API in your code, anyone who sees your script will be able to see and potentially use your key.
A better technique is to store your key in a file somewhere where it won’t be accidentally shared. Then you read the file in your script, saving the key to a variable. It is still unencrypted in memory, so this isn’t very secure, but at least it won’t in your code and be accidentally shared on GitHub.
The following commands will read the first line of a text file like the one shown below, and save the results to a variable.
You can also save your API key as a *.RData file with
the save() function, the same way you would save any R
object, and bring it back into R with load().
For even better security, save your API keys in your operating
system’s credential store using the keyring package.
See also: Google API Key Best Practices
