solutions
Large Queries and Caching Results
Overview
This notebook will demonstrate two ways of dealing with larger volumes of data:
- caching retrieved into a SQLite database
- downloading data as TIFFs
A functional definition of “large data” might include:
- any volume of data that you wouldn’t want to download twice
- any volume of data that might bog down (or worse, crash) the Cal-Adapt server, and annoy the system administrators
Saving Cal-Adapt Data to SQLite Database
To manage large downloads and avoid downloading data twice, caladaptR provides ca_getvals_db(). ca_getvals_db() is very similar to ca_getvals_tbl(), but saves the data into a SQLite database file on your computer as it comes in. Before making an API request, it checks to see if the data have already been downloaded, and if so skips it. If you get disconnected during a download, you can run the command again and it’ll just pick up where it left off. ca_getvals_db() returns a ‘remote tibble’, which functions very similar to a regular ‘in-memory’ tibble however it points to the SQLite database file on disk.
For additional info on working with SQLite databases, see the Article on making Large Queries.
Example: Download Daily Data for Congressional Disticts
In this example we’ll download daily temperature data for 16 Congressional Districts in the LA region.
Setup
Load caladaptR and the other package we’re going to need. (If you haven’t installed these yet, see this setup script).
library(caladaptr)
library(units)
library(ggplot2)
library(dplyr)
library(lubridate)
library(sf)
library(tidyr)
library(tmap)
library(conflicted)
conflict_prefer("filter", "dplyr", quiet = TRUE)
conflict_prefer("count", "dplyr", quiet = TRUE)
conflict_prefer("select", "dplyr", quiet = TRUE)
Import the Congressional Districts Boundaries
The LA Congressional district boundaries are saved in the ‘data’ folder as a geopackage:
cdist_la_fn <- "./data/cdistricts_la.gpkg"
file.exists(cdist_la_fn)[1] TRUEcdist_la_sf <- st_read(cdist_la_fn)Reading layer `cdistricts_la' from data source
`D:\GitHub\cal-adapt\caladaptr-res\docs\workshops\ca_intro_feb22\notebooks\data\cdistricts_la.gpkg'
using driver `GPKG'
Simple feature collection with 15 features and 7 fields
Geometry type: MULTIPOLYGON
Dimension: XY
Bounding box: xmin: -118.9449 ymin: 33.70403 xmax: -117.3529 ymax: 34.8233
Geodetic CRS: WGS 84tmap_mode("view")tmap mode set to interactive viewingtm_shape(cdist_la_sf) + tm_borders()Pro Tip:
- to use a sf object the location for an API Request, it must be in geographic coordinates (EPSG 4326)
Create the API Request
Here we use ca_loc_sf() as the location function for an API request for 30-years of modeled daily temperature data (minimum and maximum) for 4 GCMs and 2 RCPs:
cdist_la_cap <- ca_loc_sf(loc = cdist_la_sf, idfld = "geoid") %>%
ca_gcm(gcms[1:4]) %>%
ca_scenario(c("rcp45", "rcp85")) %>%
ca_period("day") %>%
ca_years(start = 2070, end = 2099) %>%
ca_cvar(c("tasmin", "tasmax")) %>%
ca_options(spatial_ag = "mean")
cdist_la_capCal-Adapt API Request
Location(s):
Simple Feature MULTIPOLYGON (15 feature(s))
ID field: geoid
Variable(s): tasmin, tasmax
Temporal aggregration period(s): day
GCM(s): HadGEM2-ES, CNRM-CM5, CanESM2, MIROC5
Scenario(s): rcp45, rcp85
Dates: 2070-01-01 to 2099-12-31
Options:
spatial ag: mean
Do the standard plotting and preflight checks:
plot(cdist_la_cap, locagrid = TRUE)cdist_la_cap %>% ca_preflight()General issues
- none found
Issues for querying values
- none found
Issues for downloading rasters
- none found
Fetch Data
To copy downloaded data into a database, use ca_getvals_db() instead of ca_getvals_tbl(). ca_getvals_db() has two arguments which are mandatory:
db_fn - the file name of a SQLite database (will be created if it doesn’t exist)
db_tbl - the name of a table inside the database where the values should be saved
my_database_fn <- "./data/cdist_la_temp_data.sqlite"
cdist_la_rtbl <- cdist_la_cap %>%
ca_getvals_db(db_fn = my_database_fn,
db_tbl = "temp_data",
quiet = FALSE)
|
| | 0%
|
|= | 1%
|
|== | 2%
|
|=== | 3%
|
|==== | 4%
|
|===== | 5%
|
|====== | 5%
|
|====== | 6%
|
|======= | 7%
|
|======== | 8%
|
|========= | 8%
|
|========= | 9%
|
|========== | 10%
|
|=========== | 10%
|
|=========== | 11%
|
|============ | 11%
|
|============ | 12%
|
|============= | 13%
|
|============== | 13%
|
|============== | 14%
|
|=============== | 14%
|
|=============== | 15%
|
|================ | 15%
|
|================ | 16%
|
|================= | 16%
|
|================= | 17%
|
|================== | 18%
|
|=================== | 18%
|
|=================== | 19%
|
|==================== | 19%
|
|==================== | 20%
|
|===================== | 21%
|
|====================== | 21%
|
|====================== | 22%
|
|======================= | 22%
|
|======================= | 23%
|
|======================== | 23%
|
|======================== | 24%
|
|========================= | 24%
|
|========================= | 25%
|
|========================== | 25%
|
|========================== | 26%
|
|=========================== | 26%
|
|=========================== | 27%
|
|============================ | 27%
|
|============================ | 28%
|
|============================= | 28%
|
|============================= | 29%
|
|============================== | 29%
|
|============================== | 30%
|
|=============================== | 30%
|
|=============================== | 31%
|
|================================ | 31%
|
|================================ | 32%
|
|================================= | 32%
|
|================================= | 33%
|
|================================== | 33%
|
|=================================== | 34%
|
|=================================== | 35%
|
|==================================== | 35%
|
|==================================== | 36%
|
|===================================== | 36%
|
|====================================== | 37%
|
|====================================== | 38%
|
|======================================= | 38%
|
|======================================== | 39%
|
|========================================= | 40%
|
|========================================= | 41%
|
|========================================== | 41%
|
|=========================================== | 42%
|
|============================================ | 43%
|
|============================================ | 44%
|
|============================================= | 44%
|
|============================================== | 45%
|
|=============================================== | 46%
|
|================================================ | 47%
|
|================================================= | 48%
|
|================================================== | 49%
|
|=================================================== | 50%
|
|==================================================== | 51%
|
|===================================================== | 52%
|
|====================================================== | 53%
|
|======================================================= | 54%
|
|======================================================== | 55%
|
|========================================================= | 56%
|
|========================================================== | 56%
|
|========================================================== | 57%
|
|=========================================================== | 58%
|
|============================================================ | 59%
|
|============================================================= | 59%
|
|============================================================= | 60%
|
|============================================================== | 61%
|
|=============================================================== | 62%
|
|================================================================ | 62%
|
|================================================================ | 63%
|
|================================================================= | 64%
|
|================================================================== | 64%
|
|================================================================== | 65%
|
|=================================================================== | 65%
|
|=================================================================== | 66%
|
|==================================================================== | 67%
|
|===================================================================== | 67%
|
|===================================================================== | 68%
|
|====================================================================== | 68%
|
|====================================================================== | 69%
|
|======================================================================= | 69%
|
|======================================================================= | 70%
|
|======================================================================== | 70%
|
|======================================================================== | 71%
|
|========================================================================= | 71%
|
|========================================================================= | 72%
|
|========================================================================== | 72%
|
|========================================================================== | 73%
|
|=========================================================================== | 73%
|
|=========================================================================== | 74%
|
|============================================================================ | 74%
|
|============================================================================ | 75%
|
|============================================================================= | 75%
|
|============================================================================= | 76%
|
|============================================================================== | 76%
|
|============================================================================== | 77%
|
|=============================================================================== | 77%
|
|=============================================================================== | 78%
|
|================================================================================ | 78%
|
|================================================================================ | 79%
|
|================================================================================= | 79%
|
|================================================================================== | 80%
|
|================================================================================== | 81%
|
|=================================================================================== | 81%
|
|=================================================================================== | 82%
|
|==================================================================================== | 82%
|
|===================================================================================== | 83%
|
|===================================================================================== | 84%
|
|====================================================================================== | 84%
|
|====================================================================================== | 85%
|
|======================================================================================= | 85%
|
|======================================================================================= | 86%
|
|======================================================================================== | 86%
|
|======================================================================================== | 87%
|
|========================================================================================= | 87%
|
|========================================================================================== | 88%
|
|========================================================================================== | 89%
|
|=========================================================================================== | 89%
|
|=========================================================================================== | 90%
|
|============================================================================================ | 90%
|
|============================================================================================= | 91%
|
|============================================================================================= | 92%
|
|============================================================================================== | 92%
|
|=============================================================================================== | 93%
|
|================================================================================================ | 94%
|
|================================================================================================ | 95%
|
|================================================================================================= | 95%
|
|================================================================================================== | 96%
|
|=================================================================================================== | 97%
|
|==================================================================================================== | 98%
|
|===================================================================================================== | 99%
|
|======================================================================================================| 100%
Inspect the results:
cdist_la_rtbl %>% head()
The number of rows we retrieved:
cdist_la_rtbl %>% count() %>% pull(n)[1] 2629680
Wrangling a Remote Tibble
Many of the base R operations that work with in-memory tibbles may or many not work with remote tibbles. For example as we saw above cdist_la_rtbl %>% count() works, but:
dim(cdist_la_rtbl)[1] NA 8nrow(cdist_la_rtbl)[1] NA
In general, the best way to work with remote tibbles is with
- dplyr functions, or
- SQL statements passed using the
DBIpackage
Simple filtering, sorting, grouping and simple numeric summaries generally work fine with dplyr verbs:
cdist_la_rtbl %>%
filter(geoid == "0632") %>%
group_by(scenario, gcm, cvar) %>%
summarize(mean_temp = mean(val, na.rm = TRUE))`summarise()` has grouped output by 'scenario', 'gcm'. You can override using the `.groups` argument.
Pro Tip:
- You can ‘convert’ a Remote Tibble to a regular in-memory Tibble by tacking on
collect()at the end of a dplyr expression.
If your wrangling workflow involves a lot steps that are difficult or impossible to do with remote tibbles, a good strategy is to do your filtering and grouping with dplyr statements on the remote tibble, and then convert the results to a regular tibble with collect().
Below we convert the grouped summary table into a tibble so we can use pivot_wider() (which doesn’t work on remote tibbles):
temp_long_tbl <- cdist_la_rtbl %>%
filter(geoid == "0632") %>%
group_by(scenario, gcm, cvar) %>%
summarize(mean_temp = mean(val, na.rm = TRUE)) %>%
collect()`summarise()` has grouped output by 'scenario', 'gcm'. You can override using the `.groups` argument.class(temp_long_tbl)[1] "grouped_df" "tbl_df" "tbl" "data.frame"temp_wide_tbl <- temp_long_tbl %>%
pivot_wider(names_from = cvar, values_from = mean_temp) %>%
mutate(tas_range = tasmax - tasmin)
temp_wide_tbl %>% head()
Challenge
Create a histogram of the mean daily temperatures for one Congressional District and one emissions scenario, grouping the data by GCM. Does the distribution of mean average temperature look the same across GCMs?
## Your answer here
Downloading Rasters
Another way to deal with large data needs is to download the data as raster or TIF files. The same API Request object can be used to get raster data if you feed it into ca_getrst_stars().
For additional info on downloading and analyzing rasters, see the 3 articles on Downloading Rasters.
Below we get a raster of observed historic temperature data for the Sierra climate region:
sierra_cap <- ca_loc_aoipreset(type = "climregions", idfld = "name", idval = "Sierra") %>%
ca_livneh(TRUE) %>%
ca_period("year") %>%
ca_cvar("pr") %>%
ca_years(start = 1970, end = 2010)
sierra_capCal-Adapt API Request
Location(s):
AOI Preset: climregions
name(s): Sierra
Variable(s): pr
Temporal aggregration period(s): year
Livneh data: TRUE
Dates: 1970-01-01 to 2010-12-31
plot(sierra_cap, locagrid = TRUE)
sierra_cap %>% ca_preflight()General issues
- none found
Issues for querying values
- A spatial aggregation function is required to query values from polygon areas. See `ca_options`.
Issues for downloading rasters
- none found
To fetch the data as TIFs, use :
tiff_dir <- "./data"
sierra_tiff_fn <- sierra_cap %>%
ca_getrst_stars(out_dir = tiff_dir, mask = TRUE, quiet = TRUE, overwrite = FALSE)
Pro Tip:
- to avoid downloading the same TIF multiple times, use the same output directory and set
overwrite = FALSE
ca_getrst_stars() works differently than retrieving tabular climate values. It returns a vector of TIF files that were downloaded. To work with them, you next have to load them back into R as stars objects (space-time arrays) using ca_stars_read():
sierra_stars_lst <- ca_stars_read(sierra_tiff_fn)
sierra_stars_lst[[1]]stars object with 3 dimensions and 1 attribute
attribute(s):
Min. 1st Qu. Median Mean 3rd Qu. Max. NA's
pr_year_livneh_name-Sierra 0.2009817 2.037323 2.821092 3.106411 3.863086 12.43626 123615
dimension(s):
from to offset delta refsys point values x/y
x 1 58 -121.688 0.0625 WGS 84 FALSE NULL [x]
y 1 71 40.125 -0.0625 WGS 84 FALSE NULL [y]
year 1 41 1970 1 NA NA NULL
To plot a stars objects, you have to decide which layer(s) to plot. In this case, each layer represents a year from 1970 to 2010. Below we plot 4 of the 40 years:
plot(sierra_stars_lst[[1]] %>% slice(index = seq(1,40,length.out =4), along = "year"),
axes = TRUE,
main = attributes(sierra_stars_lst[[1]])$ca_metadata$slug)
Not sure what the units are? You can double-check by viewing the metadata for the slug from the catalog:
ca_catalog_search("pr_year_livneh")
pr_year_livneh
name: Livneh yearly average precipitation historical
url: https://api.cal-adapt.org/api/series/pr_year_livneh/
tres: annual
begin: 1950-01-01T00:00:00Z
end: 2013-12-31T00:00:00Z
units: mm
num_rast: 64
id: 382
xmin: -124.5625
xmax: -113.375
ymin: 31.5625
ymax: 43.75
There is a lot more you can do with rasters, including pixel summaries, combining them into higher dimensional data cubes, spatially mosaicing them, etc. For more info, see the Rasters articles on the website.
Your Turn
Download historic precipitation data for the county where you live or work. [Answer]
## Example: Mendocino County
ca_aoipreset_geom("counties", quiet = TRUE) %>%
st_drop_geometry() %>%
filter(name == "Mendocino") %>%
select(name, state_name, fips) name state_name fips
1 Mendocino California 06045mendocino_cap <- ca_loc_aoipreset(type = "counties", idfld = "fips", idval = "06045") %>%
ca_livneh(TRUE) %>%
ca_period("year") %>%
ca_cvar("pr") %>%
ca_years(start = 1970, end = 2010)
plot(mendocino_cap)
mendocino_fn <- mendocino_cap %>%
ca_getrst_stars(out_dir = tempdir(), mask = TRUE, quiet = TRUE, overwrite = FALSE)
mendocino_stars_lst <- ca_stars_read(mendocino_fn)
mendocino_stars_lst[[1]]stars object with 3 dimensions and 1 attribute
attribute(s):
Min. 1st Qu. Median Mean 3rd Qu. Max. NA's
pr_year_livneh_fips-06045 0.9326811 2.792288 3.627976 3.892968 4.744806 11.229 7134
dimension(s):
from to offset delta refsys point values x/y
x 1 20 -124.062 0.0625 WGS 84 FALSE NULL [x]
y 1 21 40.0625 -0.0625 WGS 84 FALSE NULL [y]
year 1 41 1970 1 NA NA NULL plot(mendocino_stars_lst[[1]] %>% slice(index = seq(1,40,length.out =4), along = "year"),
axes = TRUE,
main = attributes(mendocino_stars_lst[[1]])$ca_metadata$slug)
LS0tDQp0aXRsZTogIkxhcmdlIFF1ZXJpZXMgYW5kIENhY2hpbmcgUmVzdWx0cyINCm91dHB1dDoNCiAgaHRtbF9ub3RlYm9vazogDQogICAgY3NzOiBodHRwczovL3VjYW5yLWlnaXMuZ2l0aHViLmlvL2NhbGFkYXB0ci1yZXMvYXNzZXRzL25iX2NzczAxLmNzcw0KICAgIGluY2x1ZGVzOg0KICAgICAgYmVmb3JlX2JvZHk6IGh0dHBzOi8vdWNhbnItaWdpcy5naXRodWIuaW8vY2FsYWRhcHRyLXJlcy9hc3NldHMvbmJfaGRyc29sbi5odG1sDQogICAgICBhZnRlcl9ib2R5OiBodHRwczovL3VjYW5yLWlnaXMuZ2l0aHViLmlvL2NhbGFkYXB0ci1yZXMvYXNzZXRzL25iX2Zvb3RlcjAxLmh0bWwNCi0tLQ0KDQojIE92ZXJ2aWV3DQoNClRoaXMgbm90ZWJvb2sgd2lsbCBkZW1vbnN0cmF0ZSB0d28gd2F5cyBvZiBkZWFsaW5nIHdpdGggbGFyZ2VyIHZvbHVtZXMgb2YgZGF0YToNCg0KLSBjYWNoaW5nIHJldHJpZXZlZCBpbnRvIGEgU1FMaXRlIGRhdGFiYXNlICANCi0gZG93bmxvYWRpbmcgZGF0YSBhcyBUSUZGcw0KDQpBIGZ1bmN0aW9uYWwgZGVmaW5pdGlvbiBvZiAibGFyZ2UgZGF0YSIgbWlnaHQgaW5jbHVkZToNCg0KLSBhbnkgdm9sdW1lIG9mIGRhdGEgdGhhdCB5b3Ugd291bGRuJ3Qgd2FudCB0byBkb3dubG9hZCB0d2ljZSAgDQotIGFueSB2b2x1bWUgb2YgZGF0YSB0aGF0IG1pZ2h0IGJvZyBkb3duIChvciB3b3JzZSwgY3Jhc2gpIHRoZSBDYWwtQWRhcHQgc2VydmVyLCBhbmQgYW5ub3kgdGhlIHN5c3RlbSBhZG1pbmlzdHJhdG9ycyAgDQoNCiMjIFNhdmluZyBDYWwtQWRhcHQgRGF0YSB0byBTUUxpdGUgRGF0YWJhc2UNCg0KVG8gbWFuYWdlIGxhcmdlIGRvd25sb2FkcyBhbmQgYXZvaWQgZG93bmxvYWRpbmcgZGF0YSB0d2ljZSwgY2FsYWRhcHRSIHByb3ZpZGVzIGBjYV9nZXR2YWxzX2RiKClgLiBgY2FfZ2V0dmFsc19kYigpYCBpcyB2ZXJ5IHNpbWlsYXIgdG8gYGNhX2dldHZhbHNfdGJsKClgLCBidXQgc2F2ZXMgdGhlIGRhdGEgaW50byBhIFNRTGl0ZSBkYXRhYmFzZSBmaWxlIG9uIHlvdXIgY29tcHV0ZXIgYXMgaXQgY29tZXMgaW4uIEJlZm9yZSBtYWtpbmcgYW4gQVBJIHJlcXVlc3QsIGl0IGNoZWNrcyB0byBzZWUgaWYgdGhlIGRhdGEgaGF2ZSBhbHJlYWR5IGJlZW4gZG93bmxvYWRlZCwgYW5kIGlmIHNvIHNraXBzIGl0LiBJZiB5b3UgZ2V0IGRpc2Nvbm5lY3RlZCBkdXJpbmcgYSBkb3dubG9hZCwgeW91IGNhbiBydW4gdGhlIGNvbW1hbmQgYWdhaW4gYW5kIGl0J2xsIGp1c3QgcGljayB1cCB3aGVyZSBpdCBsZWZ0IG9mZi4gYGNhX2dldHZhbHNfZGIoKWAgcmV0dXJucyBhICdyZW1vdGUgdGliYmxlJywgd2hpY2ggZnVuY3Rpb25zIHZlcnkgc2ltaWxhciB0byBhIHJlZ3VsYXIgJ2luLW1lbW9yeScgdGliYmxlIGhvd2V2ZXIgaXQgcG9pbnRzIHRvIHRoZSBTUUxpdGUgZGF0YWJhc2UgZmlsZSBvbiBkaXNrLg0KDQpGb3IgYWRkaXRpb25hbCBpbmZvIG9uIHdvcmtpbmcgd2l0aCBTUUxpdGUgZGF0YWJhc2VzLCBzZWUgdGhlIEFydGljbGUgb24gbWFraW5nIFtMYXJnZSBRdWVyaWVzXShodHRwczovL3VjYW5yLWlnaXMuZ2l0aHViLmlvL2NhbGFkYXB0ci9hcnRpY2xlcy9sYXJnZS1xdWVyaWVzLmh0bWwpLg0KDQojIEV4YW1wbGU6IERvd25sb2FkIERhaWx5IERhdGEgZm9yIENvbmdyZXNzaW9uYWwgRGlzdGljdHMNCg0KSW4gdGhpcyBleGFtcGxlIHdlJ2xsIGRvd25sb2FkIGRhaWx5IHRlbXBlcmF0dXJlIGRhdGEgZm9yIDE2IENvbmdyZXNzaW9uYWwgRGlzdHJpY3RzIGluIHRoZSBMQSByZWdpb24uIA0KDQpcDQoNCiMjIFNldHVwDQoNCkxvYWQgY2FsYWRhcHRSIGFuZCB0aGUgb3RoZXIgcGFja2FnZSB3ZSdyZSBnb2luZyB0byBuZWVkLiAoSWYgeW91IGhhdmVuJ3QgaW5zdGFsbGVkIHRoZXNlIHlldCwgc2VlIHRoaXMgW3NldHVwIHNjcmlwdF0oaHR0cHM6Ly9naXRodWIuY29tL3VjYW5yLWlnaXMvY2FsYWRhcHRyLXJlcy9ibG9iL21haW4vZG9jcy93b3Jrc2hvcHMvY2FfaW50cm9fb2N0MjEvc2NyaXB0cy9jYWxhZGFwdHJfc2V0dXAuUikpLiANCg0KYGBge3IgY2h1bmswMSwgbWVzc2FnZT1GQUxTRSwgd2FybmluZz1GQUxTRSwgcmVzdWx0cz0naG9sZCd9DQpsaWJyYXJ5KGNhbGFkYXB0cikNCmxpYnJhcnkodW5pdHMpDQpsaWJyYXJ5KGdncGxvdDIpDQpsaWJyYXJ5KGRwbHlyKQ0KbGlicmFyeShsdWJyaWRhdGUpDQpsaWJyYXJ5KHNmKQ0KbGlicmFyeSh0aWR5cikNCmxpYnJhcnkodG1hcCkNCmxpYnJhcnkoY29uZmxpY3RlZCkNCmNvbmZsaWN0X3ByZWZlcigiZmlsdGVyIiwgImRwbHlyIiwgcXVpZXQgPSBUUlVFKQ0KY29uZmxpY3RfcHJlZmVyKCJjb3VudCIsICJkcGx5ciIsIHF1aWV0ID0gVFJVRSkNCmNvbmZsaWN0X3ByZWZlcigic2VsZWN0IiwgImRwbHlyIiwgcXVpZXQgPSBUUlVFKQ0KYGBgDQoNClwNCg0KIyMgSW1wb3J0IHRoZSBDb25ncmVzc2lvbmFsIERpc3RyaWN0cyBCb3VuZGFyaWVzDQoNClRoZSBMQSBDb25ncmVzc2lvbmFsIGRpc3RyaWN0IGJvdW5kYXJpZXMgYXJlIHNhdmVkIGluIHRoZSAnZGF0YScgZm9sZGVyIGFzIGEgZ2VvcGFja2FnZToNCg0KYGBge3IgY2h1bmswMn0NCmNkaXN0X2xhX2ZuIDwtICIuL2RhdGEvY2Rpc3RyaWN0c19sYS5ncGtnIg0KZmlsZS5leGlzdHMoY2Rpc3RfbGFfZm4pDQoNCmNkaXN0X2xhX3NmIDwtIHN0X3JlYWQoY2Rpc3RfbGFfZm4pDQoNCnRtYXBfbW9kZSgidmlldyIpDQp0bV9zaGFwZShjZGlzdF9sYV9zZikgKyB0bV9ib3JkZXJzKCkNCmBgYA0KDQoqKlBybyBUaXA6KioNCg0KLSB0byB1c2UgYSBzZiBvYmplY3QgdGhlIGxvY2F0aW9uIGZvciBhbiBBUEkgUmVxdWVzdCwgaXQgbXVzdCBiZSBpbiBnZW9ncmFwaGljIGNvb3JkaW5hdGVzIChFUFNHIDQzMjYpDQoNCg0KXA0KDQojIyBDcmVhdGUgdGhlIEFQSSBSZXF1ZXN0DQoNCkhlcmUgd2UgdXNlIGBjYV9sb2Nfc2YoKWAgYXMgdGhlIGxvY2F0aW9uIGZ1bmN0aW9uIGZvciBhbiBBUEkgcmVxdWVzdCBmb3IgMzAteWVhcnMgb2YgbW9kZWxlZCBkYWlseSB0ZW1wZXJhdHVyZSBkYXRhIChtaW5pbXVtIGFuZCBtYXhpbXVtKSBmb3IgNCBHQ01zIGFuZCAyIFJDUHM6DQoNCmBgYHtyIGNodW5rMDN9DQpjZGlzdF9sYV9jYXAgPC0gY2FfbG9jX3NmKGxvYyA9IGNkaXN0X2xhX3NmLCBpZGZsZCA9ICJnZW9pZCIpICU+JSANCiAgY2FfZ2NtKGdjbXNbMTo0XSkgJT4lICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgDQogIGNhX3NjZW5hcmlvKGMoInJjcDQ1IiwgInJjcDg1IikpICU+JQ0KICBjYV9wZXJpb2QoImRheSIpICU+JQ0KICBjYV95ZWFycyhzdGFydCA9IDIwNzAsIGVuZCA9IDIwOTkpICU+JQ0KICBjYV9jdmFyKGMoInRhc21pbiIsICJ0YXNtYXgiKSkgJT4lIA0KICBjYV9vcHRpb25zKHNwYXRpYWxfYWcgPSAibWVhbiIpDQoNCmNkaXN0X2xhX2NhcA0KYGBgDQoNClwNCg0KRG8gdGhlIHN0YW5kYXJkIHBsb3R0aW5nIGFuZCBwcmVmbGlnaHQgY2hlY2tzOg0KDQpgYGB7ciBjaHVuazA0LCBjYWNoZSA9IEZBTFNFfQ0KcGxvdChjZGlzdF9sYV9jYXAsIGxvY2FncmlkID0gVFJVRSkNCmBgYA0KDQoNCmBgYHtyIGNodW5rMDV9DQpjZGlzdF9sYV9jYXAgJT4lIGNhX3ByZWZsaWdodCgpDQpgYGANCg0KXA0KDQojIyBGZXRjaCBEYXRhDQoNClRvIGNvcHkgZG93bmxvYWRlZCBkYXRhIGludG8gYSBkYXRhYmFzZSwgdXNlIGBjYV9nZXR2YWxzX2RiKClgIGluc3RlYWQgb2YgYGNhX2dldHZhbHNfdGJsKClgLiBgY2FfZ2V0dmFsc19kYigpYCBoYXMgdHdvIGFyZ3VtZW50cyB3aGljaCBhcmUgbWFuZGF0b3J5Og0KDQpgZGJfZm5gIC0gdGhlIGZpbGUgbmFtZSBvZiBhIFNRTGl0ZSBkYXRhYmFzZSAod2lsbCBiZSBjcmVhdGVkIGlmIGl0IGRvZXNuJ3QgZXhpc3QpDQoNCmBkYl90YmxgIC0gdGhlIG5hbWUgb2YgYSB0YWJsZSBpbnNpZGUgdGhlIGRhdGFiYXNlIHdoZXJlIHRoZSB2YWx1ZXMgc2hvdWxkIGJlIHNhdmVkDQoNCmBgYHtyIGNodW5rMDZ9DQpteV9kYXRhYmFzZV9mbiA8LSAiLi9kYXRhL2NkaXN0X2xhX3RlbXBfZGF0YS5zcWxpdGUiDQoNCmNkaXN0X2xhX3J0YmwgPC0gY2Rpc3RfbGFfY2FwICU+JSANCiAgY2FfZ2V0dmFsc19kYihkYl9mbiA9IG15X2RhdGFiYXNlX2ZuLCANCiAgICAgICAgICAgICAgICBkYl90YmwgPSAidGVtcF9kYXRhIiwNCiAgICAgICAgICAgICAgICBxdWlldCA9IEZBTFNFKQ0KYGBgDQoNClwNCg0KSW5zcGVjdCB0aGUgcmVzdWx0czoNCg0KYGBge3IgY2h1bmswN30NCmNkaXN0X2xhX3J0YmwgJT4lIGhlYWQoKQ0KYGBgDQoNClwNCg0KVGhlIG51bWJlciBvZiByb3dzIHdlIHJldHJpZXZlZDoNCg0KYGBge3IgY2h1bmswOH0NCmNkaXN0X2xhX3J0YmwgJT4lIGNvdW50KCkgJT4lIHB1bGwobikNCmBgYA0KDQpcDQoNCiMjIFdyYW5nbGluZyBhIFJlbW90ZSBUaWJibGUNCg0KTWFueSBvZiB0aGUgYmFzZSBSIG9wZXJhdGlvbnMgdGhhdCB3b3JrIHdpdGggaW4tbWVtb3J5IHRpYmJsZXMgbWF5IG9yIG1hbnkgbm90IHdvcmsgd2l0aCByZW1vdGUgdGliYmxlcy4gRm9yIGV4YW1wbGUgYXMgd2Ugc2F3IGFib3ZlIGBjZGlzdF9sYV9ydGJsICU+JSBjb3VudCgpYCB3b3JrcywgYnV0Og0KDQpgYGB7ciBjaHVuazA5fQ0KZGltKGNkaXN0X2xhX3J0YmwpDQpucm93KGNkaXN0X2xhX3J0YmwpDQpgYGANCg0KXA0KDQpJbiBnZW5lcmFsLCB0aGUgYmVzdCB3YXkgdG8gd29yayB3aXRoIHJlbW90ZSB0aWJibGVzIGlzIHdpdGggDQoNCmkpIGRwbHlyIGZ1bmN0aW9ucywgb3IgICANCmlpKSBTUUwgc3RhdGVtZW50cyBwYXNzZWQgdXNpbmcgdGhlIGBEQklgIHBhY2thZ2UgDQoNClNpbXBsZSBmaWx0ZXJpbmcsIHNvcnRpbmcsIGdyb3VwaW5nIGFuZCBzaW1wbGUgbnVtZXJpYyBzdW1tYXJpZXMgZ2VuZXJhbGx5IHdvcmsgZmluZSB3aXRoIGRwbHlyIHZlcmJzOg0KDQpgYGB7ciBjaHVuazEwfQ0KY2Rpc3RfbGFfcnRibCAlPiUgDQogIGZpbHRlcihnZW9pZCA9PSAiMDYzMiIpICU+JSANCiAgZ3JvdXBfYnkoc2NlbmFyaW8sIGdjbSwgY3ZhcikgJT4lIA0KICBzdW1tYXJpemUobWVhbl90ZW1wID0gbWVhbih2YWwsIG5hLnJtID0gVFJVRSkpDQpgYGANCg0KXA0KDQoqKlBybyBUaXA6KioNCg0KLSBZb3UgY2FuICdjb252ZXJ0JyBhIFJlbW90ZSBUaWJibGUgdG8gYSByZWd1bGFyIGluLW1lbW9yeSBUaWJibGUgYnkgdGFja2luZyBvbiBgY29sbGVjdCgpYCBhdCB0aGUgZW5kIG9mIGEgZHBseXIgZXhwcmVzc2lvbi4NCg0KXA0KDQpJZiB5b3VyIHdyYW5nbGluZyB3b3JrZmxvdyBpbnZvbHZlcyBhIGxvdCBzdGVwcyB0aGF0IGFyZSBkaWZmaWN1bHQgb3IgaW1wb3NzaWJsZSB0byBkbyB3aXRoIHJlbW90ZSB0aWJibGVzLCBhIGdvb2Qgc3RyYXRlZ3kgaXMgdG8gZG8geW91ciBmaWx0ZXJpbmcgYW5kIGdyb3VwaW5nIHdpdGggZHBseXIgc3RhdGVtZW50cyBvbiB0aGUgcmVtb3RlIHRpYmJsZSwgYW5kIHRoZW4gY29udmVydCB0aGUgcmVzdWx0cyB0byBhIHJlZ3VsYXIgdGliYmxlIHdpdGggYGNvbGxlY3QoKWAuDQoNCkJlbG93IHdlIGNvbnZlcnQgdGhlIGdyb3VwZWQgc3VtbWFyeSB0YWJsZSBpbnRvIGEgdGliYmxlIHNvIHdlIGNhbiB1c2UgYHBpdm90X3dpZGVyKClgICh3aGljaCBkb2Vzbid0IHdvcmsgb24gcmVtb3RlIHRpYmJsZXMpOg0KDQpgYGB7ciBjaHVuazExfQ0KdGVtcF9sb25nX3RibCA8LSBjZGlzdF9sYV9ydGJsICU+JSANCiAgZmlsdGVyKGdlb2lkID09ICIwNjMyIikgJT4lIA0KICBncm91cF9ieShzY2VuYXJpbywgZ2NtLCBjdmFyKSAlPiUgDQogIHN1bW1hcml6ZShtZWFuX3RlbXAgPSBtZWFuKHZhbCwgbmEucm0gPSBUUlVFKSkgJT4lIA0KICBjb2xsZWN0KCkNCg0KY2xhc3ModGVtcF9sb25nX3RibCkNCg0KdGVtcF93aWRlX3RibCA8LSB0ZW1wX2xvbmdfdGJsICU+JSANCiAgcGl2b3Rfd2lkZXIobmFtZXNfZnJvbSA9IGN2YXIsIHZhbHVlc19mcm9tID0gbWVhbl90ZW1wKSAlPiUgDQogIG11dGF0ZSh0YXNfcmFuZ2UgPSB0YXNtYXggLSB0YXNtaW4pDQoNCnRlbXBfd2lkZV90YmwgJT4lIGhlYWQoKQ0KYGBgDQoNClwNCg0KIyMjIENoYWxsZW5nZQ0KDQpDcmVhdGUgYSBoaXN0b2dyYW0gb2YgdGhlIG1lYW4gZGFpbHkgdGVtcGVyYXR1cmVzIGZvciBvbmUgQ29uZ3Jlc3Npb25hbCBEaXN0cmljdCBhbmQgb25lIGVtaXNzaW9ucyBzY2VuYXJpbywgZ3JvdXBpbmcgdGhlIGRhdGEgYnkgR0NNLiBEb2VzIHRoZSBkaXN0cmlidXRpb24gb2YgbWVhbiBhdmVyYWdlIHRlbXBlcmF0dXJlIGxvb2sgdGhlIHNhbWUgYWNyb3NzIEdDTXM/IA0KDQpgYGB7ciBjaHVuazEyfQ0KIyMgWW91ciBhbnN3ZXIgaGVyZQ0KDQpgYGANCg0KXA0KDQojIERvd25sb2FkaW5nIFJhc3RlcnMNCg0KQW5vdGhlciB3YXkgdG8gZGVhbCB3aXRoIGxhcmdlIGRhdGEgbmVlZHMgaXMgdG8gZG93bmxvYWQgdGhlIGRhdGEgYXMgcmFzdGVyIG9yIFRJRiBmaWxlcy4gVGhlIHNhbWUgQVBJIFJlcXVlc3Qgb2JqZWN0IGNhbiBiZSB1c2VkIHRvIGdldCByYXN0ZXIgZGF0YSBpZiB5b3UgZmVlZCBpdCBpbnRvIGBjYV9nZXRyc3Rfc3RhcnMoKWAuDQoNCkZvciBhZGRpdGlvbmFsIGluZm8gb24gZG93bmxvYWRpbmcgYW5kIGFuYWx5emluZyByYXN0ZXJzLCBzZWUgdGhlIDMgYXJ0aWNsZXMgb24gW0Rvd25sb2FkaW5nIFJhc3RlcnNdKGh0dHBzOi8vdWNhbnItaWdpcy5naXRodWIuaW8vY2FsYWRhcHRyL2FydGljbGVzL3Jhc3RlcnMtcHQxLmh0bWwpLg0KDQpCZWxvdyB3ZSBnZXQgYSByYXN0ZXIgb2Ygb2JzZXJ2ZWQgaGlzdG9yaWMgdGVtcGVyYXR1cmUgZGF0YSBmb3IgdGhlIFNpZXJyYSBjbGltYXRlIHJlZ2lvbjoNCg0KYGBge3IgY2h1bmsxMywgY2FjaGUgPSBGQUxTRX0NCnNpZXJyYV9jYXAgPC0gY2FfbG9jX2FvaXByZXNldCh0eXBlID0gImNsaW1yZWdpb25zIiwgaWRmbGQgPSAibmFtZSIsIGlkdmFsID0gIlNpZXJyYSIpICU+JSANCiAgY2FfbGl2bmVoKFRSVUUpICU+JSANCiAgY2FfcGVyaW9kKCJ5ZWFyIikgJT4lIA0KICBjYV9jdmFyKCJwciIpICU+JSANCiAgY2FfeWVhcnMoc3RhcnQgPSAxOTcwLCBlbmQgPSAyMDEwKQ0KDQpzaWVycmFfY2FwDQoNCnBsb3Qoc2llcnJhX2NhcCwgbG9jYWdyaWQgPSBUUlVFKQ0KDQpzaWVycmFfY2FwICU+JSBjYV9wcmVmbGlnaHQoKQ0KYGBgDQoNClwNCg0KVG8gZmV0Y2ggdGhlIGRhdGEgYXMgVElGcywgdXNlIDoNCg0KYGBge3IgY2h1bmsxNH0NCnRpZmZfZGlyIDwtICIuL2RhdGEiDQoNCnNpZXJyYV90aWZmX2ZuIDwtIHNpZXJyYV9jYXAgJT4lIA0KICBjYV9nZXRyc3Rfc3RhcnMob3V0X2RpciA9IHRpZmZfZGlyLCBtYXNrID0gVFJVRSwgcXVpZXQgPSBUUlVFLCBvdmVyd3JpdGUgPSBGQUxTRSkNCmBgYA0KDQpcDQoNCioqUHJvIFRpcDoqKg0KDQogLSB0byBhdm9pZCBkb3dubG9hZGluZyB0aGUgc2FtZSBUSUYgbXVsdGlwbGUgdGltZXMsIHVzZSB0aGUgc2FtZSBvdXRwdXQgZGlyZWN0b3J5IGFuZCBzZXQgYG92ZXJ3cml0ZSA9IEZBTFNFYA0KIA0KXA0KIA0KYGNhX2dldHJzdF9zdGFycygpYCB3b3JrcyBkaWZmZXJlbnRseSB0aGFuIHJldHJpZXZpbmcgdGFidWxhciBjbGltYXRlIHZhbHVlcy4gSXQgcmV0dXJucyBhIHZlY3RvciBvZiBUSUYgZmlsZXMgdGhhdCB3ZXJlIGRvd25sb2FkZWQuIFRvIHdvcmsgd2l0aCB0aGVtLCB5b3UgbmV4dCBoYXZlIHRvIGxvYWQgdGhlbSBiYWNrIGludG8gUiBhcyBzdGFycyBvYmplY3RzIChzcGFjZS10aW1lIGFycmF5cykgdXNpbmcgYGNhX3N0YXJzX3JlYWQoKWA6DQoNCmBgYHtyIGNodW5rMTUsIHBhZ2VkLnByaW50ID0gRkFMU0V9DQpzaWVycmFfc3RhcnNfbHN0IDwtIGNhX3N0YXJzX3JlYWQoc2llcnJhX3RpZmZfZm4pDQpzaWVycmFfc3RhcnNfbHN0W1sxXV0NCmBgYA0KDQpcDQoNClRvIHBsb3QgYSBzdGFycyBvYmplY3RzLCB5b3UgaGF2ZSB0byBkZWNpZGUgd2hpY2ggbGF5ZXIocykgdG8gcGxvdC4gSW4gdGhpcyBjYXNlLCBlYWNoIGxheWVyIHJlcHJlc2VudHMgYSB5ZWFyIGZyb20gMTk3MCB0byAyMDEwLiBCZWxvdyB3ZSBwbG90IDQgb2YgdGhlIDQwIHllYXJzOg0KDQpgYGB7ciBjaHVuazE2LCBjYWNoZSA9IEZBTFNFfQ0KcGxvdChzaWVycmFfc3RhcnNfbHN0W1sxXV0gJT4lIHNsaWNlKGluZGV4ID0gc2VxKDEsNDAsbGVuZ3RoLm91dCA9NCksIGFsb25nID0gInllYXIiKSwgDQogICAgIGF4ZXMgPSBUUlVFLA0KICAgICBtYWluID0gYXR0cmlidXRlcyhzaWVycmFfc3RhcnNfbHN0W1sxXV0pJGNhX21ldGFkYXRhJHNsdWcpDQpgYGANCg0KTm90IHN1cmUgd2hhdCB0aGUgdW5pdHMgYXJlPyBZb3UgY2FuIGRvdWJsZS1jaGVjayBieSB2aWV3aW5nIHRoZSBtZXRhZGF0YSBmb3IgdGhlIHNsdWcgZnJvbSB0aGUgY2F0YWxvZzoNCg0KYGBge3IgY2h1bmsxN30NCmNhX2NhdGFsb2dfc2VhcmNoKCJwcl95ZWFyX2xpdm5laCIpDQpgYGANCg0KXA0KDQpUaGVyZSBpcyBhICoqbG90KiogbW9yZSB5b3UgY2FuIGRvIHdpdGggcmFzdGVycywgaW5jbHVkaW5nIHBpeGVsIHN1bW1hcmllcywgY29tYmluaW5nIHRoZW0gaW50byBoaWdoZXIgZGltZW5zaW9uYWwgZGF0YSBjdWJlcywgc3BhdGlhbGx5IG1vc2FpY2luZyB0aGVtLCBldGMuIEZvciBtb3JlIGluZm8sIHNlZSB0aGUgUmFzdGVycyBhcnRpY2xlcyBvbiB0aGUgW3dlYnNpdGVdKGh0dHBzOi8vdWNhbnItaWdpcy5naXRodWIuaW8vY2FsYWRhcHRyLykuDQogDQpcDQoNCiMjIyMgWW91ciBUdXJuDQoNCkRvd25sb2FkIGhpc3RvcmljIHByZWNpcGl0YXRpb24gZGF0YSBmb3IgdGhlIGNvdW50eSB3aGVyZSB5b3UgbGl2ZSBvciB3b3JrLiBbW0Fuc3dlcl0oaHR0cHM6Ly9iaXQubHkvM203QmNXTSldDQogDQpgYGB7ciBjaHVuazE4LCBwYWdlZC5wcmludCA9IEZBTFNFLCBjYWNoZT1GQUxTRX0NCiMjIEV4YW1wbGU6IE1lbmRvY2lubyBDb3VudHkNCg0KY2FfYW9pcHJlc2V0X2dlb20oImNvdW50aWVzIiwgcXVpZXQgPSBUUlVFKSAlPiUgDQogIHN0X2Ryb3BfZ2VvbWV0cnkoKSAlPiUgDQogIGZpbHRlcihuYW1lID09ICJNZW5kb2Npbm8iKSAlPiUgDQogIHNlbGVjdChuYW1lLCBzdGF0ZV9uYW1lLCBmaXBzKQ0KDQptZW5kb2Npbm9fY2FwIDwtIGNhX2xvY19hb2lwcmVzZXQodHlwZSA9ICJjb3VudGllcyIsIGlkZmxkID0gImZpcHMiLCBpZHZhbCA9ICIwNjA0NSIpICU+JSANCiAgY2FfbGl2bmVoKFRSVUUpICU+JSANCiAgY2FfcGVyaW9kKCJ5ZWFyIikgJT4lIA0KICBjYV9jdmFyKCJwciIpICU+JSANCiAgY2FfeWVhcnMoc3RhcnQgPSAxOTcwLCBlbmQgPSAyMDEwKQ0KDQpwbG90KG1lbmRvY2lub19jYXApDQoNCm1lbmRvY2lub19mbiA8LSBtZW5kb2Npbm9fY2FwICU+JSANCiAgY2FfZ2V0cnN0X3N0YXJzKG91dF9kaXIgPSB0ZW1wZGlyKCksIG1hc2sgPSBUUlVFLCBxdWlldCA9IFRSVUUsIG92ZXJ3cml0ZSA9IEZBTFNFKQ0KDQptZW5kb2Npbm9fc3RhcnNfbHN0IDwtIGNhX3N0YXJzX3JlYWQobWVuZG9jaW5vX2ZuKSANCiAgDQptZW5kb2Npbm9fc3RhcnNfbHN0W1sxXV0NCg0KcGxvdChtZW5kb2Npbm9fc3RhcnNfbHN0W1sxXV0gJT4lIHNsaWNlKGluZGV4ID0gc2VxKDEsNDAsbGVuZ3RoLm91dCA9NCksIGFsb25nID0gInllYXIiKSwgDQogICAgIGF4ZXMgPSBUUlVFLA0KICAgICBtYWluID0gYXR0cmlidXRlcyhtZW5kb2Npbm9fc3RhcnNfbHN0W1sxXV0pJGNhX21ldGFkYXRhJHNsdWcpDQpgYGANCiANCg0K
