We are happy to announce a new release of our ecmwfr
R package, version 1.4.0. This new release comes with some exciting features thanks to a contribution by Elio Campitelli! Below you find a simple worked example on how to combine archetypes and batch requests to speed up data retrieval.
Batch query support example
As a pre-requisite you need to register for the ECMWF services and set your login key using the wf_set_key()
function. Once done we can proceed.
# load the library
library(ecmwfr)
# set your CDS key
wf_set_key(
user = "1111",
key = "asfasdf-128242349-asdfasdf-asdfad",
service = "cds"
)
First, we will define an ‘archetype’, a function which allows you to manipulate certain fields in a data query. In this case we tag the area
and day
fields to be dynamic. As such, these fields can be changed dynamically using the newly created dynamic_request()
function.
# this is an example of a request
# using an archetype
dynamic_request <- wf_archetype(
request = list(
"dataset_short_name" = "reanalysis-era5-pressure-levels",
"product_type" = "reanalysis",
"variable" = "temperature",
"pressure_level" = "850",
"year" = "2000",
"month" = "04",
"day" = "04",
"time" = "00:00",
"area" = "70/-20/30/60",
"format" = "netcdf",
"target" = "era5-demo.nc"
),
dynamic_fields = c("area","day"))
We use this new archetype to generate a list of two requests for two days. The new batch request function uses a list of requests during batch processing. These can be generated as shown or by substitution using a simple lapply()
routine.
# creating a list of requests using wf_archetype()
# setting the day value dynamically
batch_request <- list(
dynamic_request(day = "01"),
dynamic_request(day = "02")
)
With this new list of requests we can use the wf_request_batch()
function to request these datasets in parallel. Once the data is available a download is triggered and the data is downloaded to a desired location (with the default being temporary file storage).
# submit a batch job using 2 workers
# one for each in the list (the number of workers
# can't exceed 20)
wf_request_batch(
batch_request,
workers = 2,
user = "1111"
)