solutions 

Overview

This notebook will demonstrate two ways of dealing with larger volumes of data by caching retrieved into a local SQLite database.

A functional definition of “large data” might include:

Saving Cal-Adapt Data to SQLite Database

To manage large downloads and avoid downloading data twice, caladaptR provides ca_getvals_db(). ca_getvals_db() is very similar to ca_getvals_tbl(), but saves the data into a SQLite database file on your computer as it comes in. Before making an API request, it checks to see if the data have already been downloaded, and if so skips it. If you get disconnected during a download, you can run the command again and it’ll just pick up where it left off. ca_getvals_db() returns a ‘remote tibble’, which functions very similar to a regular ‘in-memory’ tibble however it points to the SQLite database file on disk.

For additional info on working with SQLite databases, see the Article on making Large Queries.

Example: Download Daily Data for Congressional Disticts

In this example we’ll download daily temperature data for 16 Congressional Districts in the LA region.


Setup

Load caladaptR and the other package we’re going to need. (If you haven’t installed these yet, see this setup script).

library(caladaptr)
Registered S3 methods overwritten by 'dbplyr':
  method         from
  print.tbl_lazy     
  print.tbl_sql      
Registered S3 methods overwritten by 'htmltools':
  method               from         
  print.html           tools:rstudio
  print.shiny.tag      tools:rstudio
  print.shiny.tag.list tools:rstudio
Registered S3 method overwritten by 'htmlwidgets':
  method           from         
  print.htmlwidget tools:rstudio
caladaptr (version 0.6.6)
URL: https://ucanr-igis.github.io/caladaptr
Bug reports: https://github.com/ucanr-igis/caladaptr/issues
library(units)
udunits database from C:/Users/Andy/Documents/R/win-library/4.1/units/share/udunits/udunits2.xml
library(ggplot2)
library(dplyr)

Attaching package: ‘dplyr’

The following objects are masked from ‘package:stats’:

    filter, lag

The following objects are masked from ‘package:base’:

    intersect, setdiff, setequal, union
library(lubridate)

Attaching package: ‘lubridate’

The following objects are masked from ‘package:base’:

    date, intersect, setdiff, union
library(sf)
Linking to GEOS 3.9.1, GDAL 3.2.1, PROJ 7.2.1; sf_use_s2() is TRUE
library(tidyr)
library(tmap)
library(conflicted)
conflict_prefer("filter", "dplyr", quiet = TRUE)
conflict_prefer("count", "dplyr", quiet = TRUE)
conflict_prefer("select", "dplyr", quiet = TRUE)


Import the Congressional Districts Boundaries

The LA Congressional district boundaries are saved in the ‘data’ folder as a geopackage:

cdist_la_fn <- "./data/cdistricts_la.gpkg"
file.exists(cdist_la_fn)
[1] TRUE
cdist_la_sf <- st_read(cdist_la_fn)
Reading layer `cdistricts_la' from data source 
  `D:\GitHub\cal-adapt\caladaptr-res\docs\workshops\ca_intro_apr22\notebooks\data\cdistricts_la.gpkg' 
  using driver `GPKG'
Simple feature collection with 15 features and 7 fields
Geometry type: MULTIPOLYGON
Dimension:     XY
Bounding box:  xmin: -118.9449 ymin: 33.70403 xmax: -117.3529 ymax: 34.8233
Geodetic CRS:  WGS 84
tmap_mode("view")
tmap mode set to interactive viewing
tm_shape(cdist_la_sf) + tm_borders()

Pro Tip:

  • to use a sf object the location for an API Request, it must be in geographic coordinates (EPSG 4326)


Create the API Request

Here we use ca_loc_sf() as the location function for an API request for 30-years of modeled daily temperature data (minimum and maximum) for 4 GCMs and 2 RCPs:

cdist_la_cap <- ca_loc_sf(loc = cdist_la_sf, idfld = "geoid") %>% 
  ca_gcm(gcms[1:4]) %>%                                 
  ca_scenario(c("rcp45", "rcp85")) %>%
  ca_period("day") %>%
  ca_years(start = 2070, end = 2099) %>%
  ca_cvar(c("tasmin", "tasmax")) %>% 
  ca_options(spatial_ag = "mean")

cdist_la_cap
Cal-Adapt API Request
Location(s): 
  Simple Feature MULTIPOLYGON (15 feature(s))
  ID field: geoid
Variable(s): tasmin, tasmax
Temporal aggregration period(s): day
GCM(s): HadGEM2-ES, CNRM-CM5, CanESM2, MIROC5
Scenario(s): rcp45, rcp85
Dates: 2070-01-01 to 2099-12-31
Options:
  spatial ag: mean
 


Do the standard plotting and preflight checks:

plot(cdist_la_cap, locagrid = TRUE)
cdist_la_cap %>% ca_preflight()
General issues
 - none found
Issues for querying values
 - none found
Issues for downloading rasters
 - none found


Fetch Data

To copy downloaded data into a database, use ca_getvals_db() instead of ca_getvals_tbl(). ca_getvals_db() has two arguments which are mandatory:

db_fn - the file name of a SQLite database (will be created if it doesn’t exist)

db_tbl - the name of a table inside the database where the values should be saved

my_database_fn <- "./data/cdist_la_temp_data.sqlite"

cdist_la_rtbl <- cdist_la_cap %>% 
  ca_getvals_db(db_fn = my_database_fn, 
                db_tbl = "temp_data",
                quiet = FALSE)

  |                                                                                                                
  |                                                                                                          |   0%
  |                                                                                                                
  |=                                                                                                         |   1%
  |                                                                                                                
  |==                                                                                                        |   2%
  |                                                                                                                
  |===                                                                                                       |   3%
  |                                                                                                                
  |====                                                                                                      |   3%
  |                                                                                                                
  |====                                                                                                      |   4%
  |                                                                                                                
  |=====                                                                                                     |   5%
  |                                                                                                                
  |======                                                                                                    |   5%
  |                                                                                                                
  |======                                                                                                    |   6%
  |                                                                                                                
  |=======                                                                                                   |   6%
  |                                                                                                                
  |=======                                                                                                   |   7%
  |                                                                                                                
  |========                                                                                                  |   7%
  |                                                                                                                
  |========                                                                                                  |   8%
  |                                                                                                                
  |=========                                                                                                 |   8%
  |                                                                                                                
  |=========                                                                                                 |   9%
  |                                                                                                                
  |==========                                                                                                |   9%
  |                                                                                                                
  |==========                                                                                                |  10%
  |                                                                                                                
  |===========                                                                                               |  10%
  |                                                                                                                
  |============                                                                                              |  11%
  |                                                                                                                
  |============                                                                                              |  12%
  |                                                                                                                
  |=============                                                                                             |  12%
  |                                                                                                                
  |=============                                                                                             |  13%
  |                                                                                                                
  |==============                                                                                            |  13%
  |                                                                                                                
  |===============                                                                                           |  14%
  |                                                                                                                
  |================                                                                                          |  15%
  |                                                                                                                
  |=================                                                                                         |  16%
  |                                                                                                                
  |==================                                                                                        |  17%
  |                                                                                                                
  |===================                                                                                       |  18%
  |                                                                                                                
  |====================                                                                                      |  18%
  |                                                                                                                
  |====================                                                                                      |  19%
  |                                                                                                                
  |=====================                                                                                     |  20%
  |                                                                                                                
  |======================                                                                                    |  21%
  |                                                                                                                
  |=======================                                                                                   |  21%
  |                                                                                                                
  |=======================                                                                                   |  22%
  |                                                                                                                
  |========================                                                                                  |  22%
  |                                                                                                                
  |========================                                                                                  |  23%
  |                                                                                                                
  |=========================                                                                                 |  23%
  |                                                                                                                
  |=========================                                                                                 |  24%
  |                                                                                                                
  |==========================                                                                                |  24%
  |                                                                                                                
  |==========================                                                                                |  25%
  |                                                                                                                
  |===========================                                                                               |  25%
  |                                                                                                                
  |===========================                                                                               |  26%
  |                                                                                                                
  |============================                                                                              |  26%
  |                                                                                                                
  |============================                                                                              |  27%
  |                                                                                                                
  |=============================                                                                             |  27%
  |                                                                                                                
  |=============================                                                                             |  28%
  |                                                                                                                
  |==============================                                                                            |  28%
  |                                                                                                                
  |===============================                                                                           |  29%
  |                                                                                                                
  |===============================                                                                           |  30%
  |                                                                                                                
  |================================                                                                          |  30%
  |                                                                                                                
  |================================                                                                          |  31%
  |                                                                                                                
  |=================================                                                                         |  31%
  |                                                                                                                
  |==================================                                                                        |  32%
  |                                                                                                                
  |===================================                                                                       |  33%
  |                                                                                                                
  |====================================                                                                      |  34%
  |                                                                                                                
  |=====================================                                                                     |  35%
  |                                                                                                                
  |======================================                                                                    |  36%
  |                                                                                                                
  |=======================================                                                                   |  36%
  |                                                                                                                
  |=======================================                                                                   |  37%
  |                                                                                                                
  |========================================                                                                  |  38%
  |                                                                                                                
  |=========================================                                                                 |  38%
  |                                                                                                                
  |=========================================                                                                 |  39%
  |                                                                                                                
  |==========================================                                                                |  39%
  |                                                                                                                
  |==========================================                                                                |  40%
  |                                                                                                                
  |===========================================                                                               |  40%
  |                                                                                                                
  |===========================================                                                               |  41%
  |                                                                                                                
  |============================================                                                              |  41%
  |                                                                                                                
  |============================================                                                              |  42%
  |                                                                                                                
  |=============================================                                                             |  42%
  |                                                                                                                
  |=============================================                                                             |  43%
  |                                                                                                                
  |==============================================                                                            |  43%
  |                                                                                                                
  |==============================================                                                            |  44%
  |                                                                                                                
  |===============================================                                                           |  44%
  |                                                                                                                
  |===============================================                                                           |  45%
  |                                                                                                                
  |================================================                                                          |  45%
  |                                                                                                                
  |================================================                                                          |  46%
  |                                                                                                                
  |=================================================                                                         |  46%
  |                                                                                                                
  |==================================================                                                        |  47%
  |                                                                                                                
  |===================================================                                                       |  48%
  |                                                                                                                
  |===================================================                                                       |  49%
  |                                                                                                                
  |====================================================                                                      |  49%
  |                                                                                                                
  |=====================================================                                                     |  50%
  |                                                                                                                
  |======================================================                                                    |  51%
  |                                                                                                                
  |=======================================================                                                   |  51%
  |                                                                                                                
  |=======================================================                                                   |  52%
  |                                                                                                                
  |========================================================                                                  |  53%
  |                                                                                                                
  |=========================================================                                                 |  54%
  |                                                                                                                
  |==========================================================                                                |  54%
  |                                                                                                                
  |==========================================================                                                |  55%
  |                                                                                                                
  |===========================================================                                               |  55%
  |                                                                                                                
  |===========================================================                                               |  56%
  |                                                                                                                
  |============================================================                                              |  56%
  |                                                                                                                
  |============================================================                                              |  57%
  |                                                                                                                
  |=============================================================                                             |  57%
  |                                                                                                                
  |=============================================================                                             |  58%
  |                                                                                                                
  |==============================================================                                            |  58%
  |                                                                                                                
  |==============================================================                                            |  59%
  |                                                                                                                
  |===============================================================                                           |  59%
  |                                                                                                                
  |===============================================================                                           |  60%
  |                                                                                                                
  |================================================================                                          |  60%
  |                                                                                                                
  |================================================================                                          |  61%
  |                                                                                                                
  |=================================================================                                         |  61%
  |                                                                                                                
  |=================================================================                                         |  62%
  |                                                                                                                
  |==================================================================                                        |  62%
  |                                                                                                                
  |===================================================================                                       |  63%
  |                                                                                                                
  |===================================================================                                       |  64%
  |                                                                                                                
  |====================================================================                                      |  64%
  |                                                                                                                
  |=====================================================================                                     |  65%
  |                                                                                                                
  |======================================================================                                    |  66%
  |                                                                                                                
  |=======================================================================                                   |  67%
  |                                                                                                                
  |========================================================================                                  |  68%
  |                                                                                                                
  |=========================================================================                                 |  69%
  |                                                                                                                
  |==========================================================================                                |  69%
  |                                                                                                                
  |==========================================================================                                |  70%
  |                                                                                                                
  |===========================================================================                               |  70%
  |                                                                                                                
  |===========================================================================                               |  71%
  |                                                                                                                
  |============================================================================                              |  72%
  |                                                                                                                
  |=============================================================================                             |  72%
  |                                                                                                                
  |=============================================================================                             |  73%
  |                                                                                                                
  |==============================================================================                            |  73%
  |                                                                                                                
  |==============================================================================                            |  74%
  |                                                                                                                
  |===============================================================================                           |  74%
  |                                                                                                                
  |===============================================================================                           |  75%
  |                                                                                                                
  |================================================================================                          |  75%
  |                                                                                                                
  |================================================================================                          |  76%
  |                                                                                                                
  |=================================================================================                         |  76%
  |                                                                                                                
  |=================================================================================                         |  77%
  |                                                                                                                
  |==================================================================================                        |  77%
  |                                                                                                                
  |==================================================================================                        |  78%
  |                                                                                                                
  |===================================================================================                       |  78%
  |                                                                                                                
  |===================================================================================                       |  79%
  |                                                                                                                
  |====================================================================================                      |  79%
  |                                                                                                                
  |=====================================================================================                     |  80%
  |                                                                                                                
  |======================================================================================                    |  81%
  |                                                                                                                
  |======================================================================================                    |  82%
  |                                                                                                                
  |=======================================================================================                   |  82%
  |                                                                                                                
  |========================================================================================                  |  83%
  |                                                                                                                
  |=========================================================================================                 |  84%
  |                                                                                                                
  |==========================================================================================                |  85%
  |                                                                                                                
  |===========================================================================================               |  86%
  |                                                                                                                
  |============================================================================================              |  87%
  |                                                                                                                
  |=============================================================================================             |  87%
  |                                                                                                                
  |=============================================================================================             |  88%
  |                                                                                                                
  |==============================================================================================            |  88%
  |                                                                                                                
  |==============================================================================================            |  89%
  |                                                                                                                
  |===============================================================================================           |  90%
  |                                                                                                                
  |================================================================================================          |  90%
  |                                                                                                                
  |================================================================================================          |  91%
  |                                                                                                                
  |=================================================================================================         |  91%
  |                                                                                                                
  |=================================================================================================         |  92%
  |                                                                                                                
  |==================================================================================================        |  92%
  |                                                                                                                
  |==================================================================================================        |  93%
  |                                                                                                                
  |===================================================================================================       |  93%
  |                                                                                                                
  |===================================================================================================       |  94%
  |                                                                                                                
  |====================================================================================================      |  94%
  |                                                                                                                
  |====================================================================================================      |  95%
  |                                                                                                                
  |=====================================================================================================     |  95%
  |                                                                                                                
  |======================================================================================================    |  96%
  |                                                                                                                
  |======================================================================================================    |  97%
  |                                                                                                                
  |=======================================================================================================   |  97%
  |                                                                                                                
  |========================================================================================================  |  98%
  |                                                                                                                
  |========================================================================================================= |  99%
  |                                                                                                                
  |==========================================================================================================| 100%


Inspect the results:

cdist_la_rtbl %>% head()


The number of rows we retrieved:

cdist_la_rtbl %>% count() %>% pull(n)
[1] 2629680


Wrangling a Remote Tibble

Many of the base R operations that work with in-memory tibbles may or many not work with remote tibbles. For example as we saw above cdist_la_rtbl %>% count() works, but:

dim(cdist_la_rtbl)
[1] NA  8
nrow(cdist_la_rtbl)
[1] NA


In general, the best way to work with remote tibbles is with

  1. dplyr functions, or
  2. SQL statements passed using the DBI package

Simple filtering, sorting, grouping and simple numeric summaries generally work fine with dplyr verbs:

cdist_la_rtbl %>% 
  filter(geoid == "0632") %>% 
  group_by(scenario, gcm, cvar) %>% 
  summarize(mean_temp = mean(val, na.rm = TRUE))
`summarise()` has grouped output by 'scenario', 'gcm'. You can override using the `.groups` argument.


Pro Tip:

  • You can ‘convert’ a Remote Tibble to a regular in-memory Tibble by tacking on collect() at the end of a dplyr expression.


If your wrangling workflow involves a lot steps that are difficult or impossible to do with remote tibbles, a good strategy is to do your filtering and grouping with dplyr statements on the remote tibble, and then convert the results to a regular tibble with collect().

Below we convert the grouped summary table into a tibble so we can use pivot_wider() (which doesn’t work on remote tibbles):

temp_long_tbl <- cdist_la_rtbl %>% 
  filter(geoid == "0632") %>% 
  group_by(scenario, gcm, cvar) %>% 
  summarize(mean_temp = mean(val, na.rm = TRUE)) %>% 
  collect()
`summarise()` has grouped output by 'scenario', 'gcm'. You can override using the `.groups` argument.
class(temp_long_tbl)
[1] "grouped_df" "tbl_df"     "tbl"        "data.frame"
temp_wide_tbl <- temp_long_tbl %>% 
  pivot_wider(names_from = cvar, values_from = mean_temp) %>% 
  mutate(tas_range = tasmax - tasmin)

temp_wide_tbl %>% head()


Challenge

Create a histogram of the mean daily temperatures for one Congressional District and one emissions scenario, grouping the data by GCM. Does the distribution of mean average temperature look the same across GCMs?

## Your answer here


LS0tDQp0aXRsZTogIkxhcmdlIFF1ZXJpZXMiDQpvdXRwdXQ6DQogIGh0bWxfbm90ZWJvb2s6IA0KICAgIGNzczogaHR0cHM6Ly91Y2Fuci1pZ2lzLmdpdGh1Yi5pby9jYWxhZGFwdHItcmVzL2Fzc2V0cy9uYl9jc3MwMS5jc3MNCiAgICBpbmNsdWRlczoNCiAgICAgIGJlZm9yZV9ib2R5OiBodHRwczovL3VjYW5yLWlnaXMuZ2l0aHViLmlvL2NhbGFkYXB0ci1yZXMvYXNzZXRzL25iX2hkcnNvbG4uaHRtbA0KICAgICAgYWZ0ZXJfYm9keTogaHR0cHM6Ly91Y2Fuci1pZ2lzLmdpdGh1Yi5pby9jYWxhZGFwdHItcmVzL2Fzc2V0cy9uYl9mb290ZXIwMS5odG1sDQotLS0NCg0KIyBPdmVydmlldw0KDQpUaGlzIG5vdGVib29rIHdpbGwgZGVtb25zdHJhdGUgdHdvIHdheXMgb2YgZGVhbGluZyB3aXRoIGxhcmdlciB2b2x1bWVzIG9mIGRhdGEgYnkgY2FjaGluZyByZXRyaWV2ZWQgaW50byBhIGxvY2FsIFNRTGl0ZSBkYXRhYmFzZS4gIA0KDQpBIGZ1bmN0aW9uYWwgZGVmaW5pdGlvbiBvZiAibGFyZ2UgZGF0YSIgbWlnaHQgaW5jbHVkZToNCg0KLSBhbnkgdm9sdW1lIG9mIGRhdGEgdGhhdCB5b3Ugd291bGRuJ3Qgd2FudCB0byBkb3dubG9hZCB0d2ljZSAgDQotIGFueSB2b2x1bWUgb2YgZGF0YSB0aGF0IG1pZ2h0IGJvZyBkb3duIChvciB3b3JzZSwgY3Jhc2gpIHRoZSBDYWwtQWRhcHQgc2VydmVyLCBhbmQgYW5ub3kgdGhlIHN5c3RlbSBhZG1pbmlzdHJhdG9ycyAgDQoNCiMjIFNhdmluZyBDYWwtQWRhcHQgRGF0YSB0byBTUUxpdGUgRGF0YWJhc2UNCg0KVG8gbWFuYWdlIGxhcmdlIGRvd25sb2FkcyBhbmQgYXZvaWQgZG93bmxvYWRpbmcgZGF0YSB0d2ljZSwgY2FsYWRhcHRSIHByb3ZpZGVzIGBjYV9nZXR2YWxzX2RiKClgLiBgY2FfZ2V0dmFsc19kYigpYCBpcyB2ZXJ5IHNpbWlsYXIgdG8gYGNhX2dldHZhbHNfdGJsKClgLCBidXQgc2F2ZXMgdGhlIGRhdGEgaW50byBhIFNRTGl0ZSBkYXRhYmFzZSBmaWxlIG9uIHlvdXIgY29tcHV0ZXIgYXMgaXQgY29tZXMgaW4uIEJlZm9yZSBtYWtpbmcgYW4gQVBJIHJlcXVlc3QsIGl0IGNoZWNrcyB0byBzZWUgaWYgdGhlIGRhdGEgaGF2ZSBhbHJlYWR5IGJlZW4gZG93bmxvYWRlZCwgYW5kIGlmIHNvIHNraXBzIGl0LiBJZiB5b3UgZ2V0IGRpc2Nvbm5lY3RlZCBkdXJpbmcgYSBkb3dubG9hZCwgeW91IGNhbiBydW4gdGhlIGNvbW1hbmQgYWdhaW4gYW5kIGl0J2xsIGp1c3QgcGljayB1cCB3aGVyZSBpdCBsZWZ0IG9mZi4gYGNhX2dldHZhbHNfZGIoKWAgcmV0dXJucyBhICdyZW1vdGUgdGliYmxlJywgd2hpY2ggZnVuY3Rpb25zIHZlcnkgc2ltaWxhciB0byBhIHJlZ3VsYXIgJ2luLW1lbW9yeScgdGliYmxlIGhvd2V2ZXIgaXQgcG9pbnRzIHRvIHRoZSBTUUxpdGUgZGF0YWJhc2UgZmlsZSBvbiBkaXNrLg0KDQpGb3IgYWRkaXRpb25hbCBpbmZvIG9uIHdvcmtpbmcgd2l0aCBTUUxpdGUgZGF0YWJhc2VzLCBzZWUgdGhlIEFydGljbGUgb24gbWFraW5nIFtMYXJnZSBRdWVyaWVzXShodHRwczovL3VjYW5yLWlnaXMuZ2l0aHViLmlvL2NhbGFkYXB0ci9hcnRpY2xlcy9sYXJnZS1xdWVyaWVzLmh0bWwpLg0KDQojIEV4YW1wbGU6IERvd25sb2FkIERhaWx5IERhdGEgZm9yIENvbmdyZXNzaW9uYWwgRGlzdGljdHMNCg0KSW4gdGhpcyBleGFtcGxlIHdlJ2xsIGRvd25sb2FkIGRhaWx5IHRlbXBlcmF0dXJlIGRhdGEgZm9yIDE2IENvbmdyZXNzaW9uYWwgRGlzdHJpY3RzIGluIHRoZSBMQSByZWdpb24uIA0KDQpcDQoNCiMjIFNldHVwDQoNCkxvYWQgY2FsYWRhcHRSIGFuZCB0aGUgb3RoZXIgcGFja2FnZSB3ZSdyZSBnb2luZyB0byBuZWVkLiAoSWYgeW91IGhhdmVuJ3QgaW5zdGFsbGVkIHRoZXNlIHlldCwgc2VlIHRoaXMgW3NldHVwIHNjcmlwdF0oaHR0cHM6Ly9naXRodWIuY29tL3VjYW5yLWlnaXMvY2FsYWRhcHRyLXJlcy9ibG9iL21haW4vZG9jcy93b3Jrc2hvcHMvY2FfaW50cm9fYXByMjIvc2NyaXB0cy9jYWxhZGFwdHJfc2V0dXAuUikpLiANCg0KYGBge3IgY2h1bmswMSwgbWVzc2FnZT1GQUxTRSwgd2FybmluZz1GQUxTRSwgcmVzdWx0cz0naG9sZCd9DQpsaWJyYXJ5KGNhbGFkYXB0cikNCmxpYnJhcnkodW5pdHMpDQpsaWJyYXJ5KGdncGxvdDIpDQpsaWJyYXJ5KGRwbHlyKQ0KbGlicmFyeShsdWJyaWRhdGUpDQpsaWJyYXJ5KHNmKQ0KbGlicmFyeSh0aWR5cikNCmxpYnJhcnkodG1hcCkNCmxpYnJhcnkoY29uZmxpY3RlZCkNCmNvbmZsaWN0X3ByZWZlcigiZmlsdGVyIiwgImRwbHlyIiwgcXVpZXQgPSBUUlVFKQ0KY29uZmxpY3RfcHJlZmVyKCJjb3VudCIsICJkcGx5ciIsIHF1aWV0ID0gVFJVRSkNCmNvbmZsaWN0X3ByZWZlcigic2VsZWN0IiwgImRwbHlyIiwgcXVpZXQgPSBUUlVFKQ0KYGBgDQoNClwNCg0KIyMgSW1wb3J0IHRoZSBDb25ncmVzc2lvbmFsIERpc3RyaWN0cyBCb3VuZGFyaWVzDQoNClRoZSBMQSBDb25ncmVzc2lvbmFsIGRpc3RyaWN0IGJvdW5kYXJpZXMgYXJlIHNhdmVkIGluIHRoZSAnZGF0YScgZm9sZGVyIGFzIGEgZ2VvcGFja2FnZToNCg0KYGBge3IgY2h1bmswMn0NCmNkaXN0X2xhX2ZuIDwtICIuL2RhdGEvY2Rpc3RyaWN0c19sYS5ncGtnIg0KZmlsZS5leGlzdHMoY2Rpc3RfbGFfZm4pDQoNCmNkaXN0X2xhX3NmIDwtIHN0X3JlYWQoY2Rpc3RfbGFfZm4pDQoNCnRtYXBfbW9kZSgidmlldyIpDQp0bV9zaGFwZShjZGlzdF9sYV9zZikgKyB0bV9ib3JkZXJzKCkNCmBgYA0KDQoqKlBybyBUaXA6KioNCg0KLSB0byB1c2UgYSBzZiBvYmplY3QgdGhlIGxvY2F0aW9uIGZvciBhbiBBUEkgUmVxdWVzdCwgaXQgbXVzdCBiZSBpbiBnZW9ncmFwaGljIGNvb3JkaW5hdGVzIChFUFNHIDQzMjYpDQoNClwNCg0KIyMgQ3JlYXRlIHRoZSBBUEkgUmVxdWVzdA0KDQpIZXJlIHdlIHVzZSBgY2FfbG9jX3NmKClgIGFzIHRoZSBsb2NhdGlvbiBmdW5jdGlvbiBmb3IgYW4gQVBJIHJlcXVlc3QgZm9yIDMwLXllYXJzIG9mIG1vZGVsZWQgZGFpbHkgdGVtcGVyYXR1cmUgZGF0YSAobWluaW11bSBhbmQgbWF4aW11bSkgZm9yIDQgR0NNcyBhbmQgMiBSQ1BzOg0KDQpgYGB7ciBjaHVuazAzfQ0KY2Rpc3RfbGFfY2FwIDwtIGNhX2xvY19zZihsb2MgPSBjZGlzdF9sYV9zZiwgaWRmbGQgPSAiZ2VvaWQiKSAlPiUgDQogIGNhX2djbShnY21zWzE6NF0pICU+JSAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIA0KICBjYV9zY2VuYXJpbyhjKCJyY3A0NSIsICJyY3A4NSIpKSAlPiUNCiAgY2FfcGVyaW9kKCJkYXkiKSAlPiUNCiAgY2FfeWVhcnMoc3RhcnQgPSAyMDcwLCBlbmQgPSAyMDk5KSAlPiUNCiAgY2FfY3ZhcihjKCJ0YXNtaW4iLCAidGFzbWF4IikpICU+JSANCiAgY2Ffb3B0aW9ucyhzcGF0aWFsX2FnID0gIm1lYW4iKQ0KDQpjZGlzdF9sYV9jYXANCmBgYA0KDQpcDQoNCkRvIHRoZSBzdGFuZGFyZCBwbG90dGluZyBhbmQgcHJlZmxpZ2h0IGNoZWNrczoNCg0KYGBge3IgY2h1bmswNCwgY2FjaGUgPSBGQUxTRX0NCnBsb3QoY2Rpc3RfbGFfY2FwLCBsb2NhZ3JpZCA9IFRSVUUpDQpgYGANCg0KDQpgYGB7ciBjaHVuazA1fQ0KY2Rpc3RfbGFfY2FwICU+JSBjYV9wcmVmbGlnaHQoKQ0KYGBgDQoNClwNCg0KIyMgRmV0Y2ggRGF0YQ0KDQpUbyBjb3B5IGRvd25sb2FkZWQgZGF0YSBpbnRvIGEgZGF0YWJhc2UsIHVzZSBgY2FfZ2V0dmFsc19kYigpYCBpbnN0ZWFkIG9mIGBjYV9nZXR2YWxzX3RibCgpYC4gYGNhX2dldHZhbHNfZGIoKWAgaGFzIHR3byBhcmd1bWVudHMgd2hpY2ggYXJlIG1hbmRhdG9yeToNCg0KYGRiX2ZuYCAtIHRoZSBmaWxlIG5hbWUgb2YgYSBTUUxpdGUgZGF0YWJhc2UgKHdpbGwgYmUgY3JlYXRlZCBpZiBpdCBkb2Vzbid0IGV4aXN0KQ0KDQpgZGJfdGJsYCAtIHRoZSBuYW1lIG9mIGEgdGFibGUgaW5zaWRlIHRoZSBkYXRhYmFzZSB3aGVyZSB0aGUgdmFsdWVzIHNob3VsZCBiZSBzYXZlZA0KDQpgYGB7ciBjaHVuazA2fQ0KbXlfZGF0YWJhc2VfZm4gPC0gIi4vZGF0YS9jZGlzdF9sYV90ZW1wX2RhdGEuc3FsaXRlIg0KDQpjZGlzdF9sYV9ydGJsIDwtIGNkaXN0X2xhX2NhcCAlPiUgDQogIGNhX2dldHZhbHNfZGIoZGJfZm4gPSBteV9kYXRhYmFzZV9mbiwgDQogICAgICAgICAgICAgICAgZGJfdGJsID0gInRlbXBfZGF0YSIsDQogICAgICAgICAgICAgICAgcXVpZXQgPSBGQUxTRSkNCmBgYA0KDQpcDQoNCkluc3BlY3QgdGhlIHJlc3VsdHM6DQoNCmBgYHtyIGNodW5rMDd9DQpjZGlzdF9sYV9ydGJsICU+JSBoZWFkKCkNCmBgYA0KDQpcDQoNClRoZSBudW1iZXIgb2Ygcm93cyB3ZSByZXRyaWV2ZWQ6DQoNCmBgYHtyIGNodW5rMDh9DQpjZGlzdF9sYV9ydGJsICU+JSBjb3VudCgpICU+JSBwdWxsKG4pDQpgYGANCg0KXA0KDQojIyBXcmFuZ2xpbmcgYSBSZW1vdGUgVGliYmxlDQoNCk1hbnkgb2YgdGhlIGJhc2UgUiBvcGVyYXRpb25zIHRoYXQgd29yayB3aXRoIGluLW1lbW9yeSB0aWJibGVzIG1heSBvciBtYW55IG5vdCB3b3JrIHdpdGggcmVtb3RlIHRpYmJsZXMuIEZvciBleGFtcGxlIGFzIHdlIHNhdyBhYm92ZSBgY2Rpc3RfbGFfcnRibCAlPiUgY291bnQoKWAgd29ya3MsIGJ1dDoNCg0KYGBge3IgY2h1bmswOX0NCmRpbShjZGlzdF9sYV9ydGJsKQ0KbnJvdyhjZGlzdF9sYV9ydGJsKQ0KYGBgDQoNClwNCg0KSW4gZ2VuZXJhbCwgdGhlIGJlc3Qgd2F5IHRvIHdvcmsgd2l0aCByZW1vdGUgdGliYmxlcyBpcyB3aXRoIA0KDQppKSBkcGx5ciBmdW5jdGlvbnMsIG9yICAgDQppaSkgU1FMIHN0YXRlbWVudHMgcGFzc2VkIHVzaW5nIHRoZSBgREJJYCBwYWNrYWdlIA0KDQpTaW1wbGUgZmlsdGVyaW5nLCBzb3J0aW5nLCBncm91cGluZyBhbmQgc2ltcGxlIG51bWVyaWMgc3VtbWFyaWVzIGdlbmVyYWxseSB3b3JrIGZpbmUgd2l0aCBkcGx5ciB2ZXJiczoNCg0KYGBge3IgY2h1bmsxMH0NCmNkaXN0X2xhX3J0YmwgJT4lIA0KICBmaWx0ZXIoZ2VvaWQgPT0gIjA2MzIiKSAlPiUgDQogIGdyb3VwX2J5KHNjZW5hcmlvLCBnY20sIGN2YXIpICU+JSANCiAgc3VtbWFyaXplKG1lYW5fdGVtcCA9IG1lYW4odmFsLCBuYS5ybSA9IFRSVUUpKQ0KYGBgDQoNClwNCg0KKipQcm8gVGlwOioqDQoNCi0gWW91IGNhbiAnY29udmVydCcgYSBSZW1vdGUgVGliYmxlIHRvIGEgcmVndWxhciBpbi1tZW1vcnkgVGliYmxlIGJ5IHRhY2tpbmcgb24gYGNvbGxlY3QoKWAgYXQgdGhlIGVuZCBvZiBhIGRwbHlyIGV4cHJlc3Npb24uDQoNClwNCg0KSWYgeW91ciB3cmFuZ2xpbmcgd29ya2Zsb3cgaW52b2x2ZXMgYSBsb3Qgc3RlcHMgdGhhdCBhcmUgZGlmZmljdWx0IG9yIGltcG9zc2libGUgdG8gZG8gd2l0aCByZW1vdGUgdGliYmxlcywgYSBnb29kIHN0cmF0ZWd5IGlzIHRvIGRvIHlvdXIgZmlsdGVyaW5nIGFuZCBncm91cGluZyB3aXRoIGRwbHlyIHN0YXRlbWVudHMgb24gdGhlIHJlbW90ZSB0aWJibGUsIGFuZCB0aGVuIGNvbnZlcnQgdGhlIHJlc3VsdHMgdG8gYSByZWd1bGFyIHRpYmJsZSB3aXRoIGBjb2xsZWN0KClgLg0KDQpCZWxvdyB3ZSBjb252ZXJ0IHRoZSBncm91cGVkIHN1bW1hcnkgdGFibGUgaW50byBhIHRpYmJsZSBzbyB3ZSBjYW4gdXNlIGBwaXZvdF93aWRlcigpYCAod2hpY2ggZG9lc24ndCB3b3JrIG9uIHJlbW90ZSB0aWJibGVzKToNCg0KYGBge3IgY2h1bmsxMX0NCnRlbXBfbG9uZ190YmwgPC0gY2Rpc3RfbGFfcnRibCAlPiUgDQogIGZpbHRlcihnZW9pZCA9PSAiMDYzMiIpICU+JSANCiAgZ3JvdXBfYnkoc2NlbmFyaW8sIGdjbSwgY3ZhcikgJT4lIA0KICBzdW1tYXJpemUobWVhbl90ZW1wID0gbWVhbih2YWwsIG5hLnJtID0gVFJVRSkpICU+JSANCiAgY29sbGVjdCgpDQoNCmNsYXNzKHRlbXBfbG9uZ190YmwpDQoNCnRlbXBfd2lkZV90YmwgPC0gdGVtcF9sb25nX3RibCAlPiUgDQogIHBpdm90X3dpZGVyKG5hbWVzX2Zyb20gPSBjdmFyLCB2YWx1ZXNfZnJvbSA9IG1lYW5fdGVtcCkgJT4lIA0KICBtdXRhdGUodGFzX3JhbmdlID0gdGFzbWF4IC0gdGFzbWluKQ0KDQp0ZW1wX3dpZGVfdGJsICU+JSBoZWFkKCkNCmBgYA0KDQpcDQoNCiMjIyBDaGFsbGVuZ2UNCg0KQ3JlYXRlIGEgaGlzdG9ncmFtIG9mIHRoZSBtZWFuIGRhaWx5IHRlbXBlcmF0dXJlcyBmb3Igb25lIENvbmdyZXNzaW9uYWwgRGlzdHJpY3QgYW5kIG9uZSBlbWlzc2lvbnMgc2NlbmFyaW8sIGdyb3VwaW5nIHRoZSBkYXRhIGJ5IEdDTS4gRG9lcyB0aGUgZGlzdHJpYnV0aW9uIG9mIG1lYW4gYXZlcmFnZSB0ZW1wZXJhdHVyZSBsb29rIHRoZSBzYW1lIGFjcm9zcyBHQ01zPyANCg0KYGBge3IgY2h1bmsxMn0NCiMjIFlvdXIgYW5zd2VyIGhlcmUNCg0KYGBgDQoNClwNCg0KDQo=