RasterDataSources.jl

RasterDataSources downloads raster data for local use or for integration into other spatial data packages, like Rasters.jl. The collection is largely focussed on datasets relevant to ecology, but will have a lot of crossover with other sciences.

Currently sources include:

Please open an issue if you need more datasets added, or (even better) open a pull request following the form of the other datasets where possible.

Retrieving data

Usage is generally via the getraster method - which will download the raster data source if it isn't available locally, or simply return the path/s of the raster file/s:

julia> using RasterDataSources

julia> getraster(WorldClim{Climate}, :wind; month=1:12)
12-element Array{String,1}:
 "/home/user/Data/WorldClim/Climate/wind/wc2.1_10m_wind_01.tif"
 "/home/user/Data/WorldClim/Climate/wind/wc2.1_10m_wind_02.tif"
 "/home/user/Data/WorldClim/Climate/wind/wc2.1_10m_wind_03.tif"
 "/home/user/Data/WorldClim/Climate/wind/wc2.1_10m_wind_04.tif"
 "/home/user/Data/WorldClim/Climate/wind/wc2.1_10m_wind_05.tif"
 "/home/user/Data/WorldClim/Climate/wind/wc2.1_10m_wind_06.tif"
 "/home/user/Data/WorldClim/Climate/wind/wc2.1_10m_wind_07.tif"
 "/home/user/Data/WorldClim/Climate/wind/wc2.1_10m_wind_08.tif"
 "/home/user/Data/WorldClim/Climate/wind/wc2.1_10m_wind_09.tif"
 "/home/user/Data/WorldClim/Climate/wind/wc2.1_10m_wind_10.tif"
 "/home/user/Data/WorldClim/Climate/wind/wc2.1_10m_wind_11.tif"
 "/home/user/Data/WorldClim/Climate/wind/wc2.1_10m_wind_12.tif"

Installation and setup

Install as usual with:

] add RasterDataSources

To download data you will need to specify a folder to put it in. You can do this by assigning the environment variable RASTERDATASOURCES_PATH:

ENV["RASTERDATASOURCES_PATH"] = "/home/user/Data/"

This can be put in your startup.jl file or the system environment.

RasterDataSources was based on code from the SimpleSDMDataSoures.jl package by Timothée Poisot.

getraster(source::Type, [layer]; kw...)

Download raster layers layers from the data source, returning a String for a single layer, or a NamedTuple for a Tuple of layers.

getraster provides a standardised interface to download data sources, and return the filename/s of the selected files.

getraster can be called with any RasterDataSource. See the docstrings of these types for the specific keywords and arguments. RasterDataSource with documented getraster usage are WorldClim, EarthEnv, CHELSA, AWAP, ALWB, and MODIS. All these implementations follow the template specified below.

Arguments

• source: defines the RasterDataSource and (if it there is more than one) the specific RasterDataSet from which to download data.
• layer: choose the named Symbol/s or numbered Int/s (for BioClim) layer/s of the data source. If layer is not passed, all layers will be downloaded, returning a NamedTuple of filenames.

Keywords

Keyword arguments specify subsets of a data set, such as by date or resolution. As much as possible these are standardised for all sources where they are relevent.

• date: DateTime date, range of dates, or tuple of start and end dates. Usually for weather datasets.
• month: month or range of months to download for climatic datasets, as Integers from 1 to 12.
• res: spatial resolion of the file, as a String with units, e.g. "10m".

Return values

The return value is either a single String, a Tuple/Array of String, or a Tuple/Array of Tuple/Array of String –- depending on the arguments. If multiple layers are specified, this may return multiple filenames. If multiple months or dates are specified, this may also return multiple filenames.

Keyword arguments depend on the specific data source. They may modify the return value, following a pattern:

• month keywords of AbstractArray will return a Vector{String} or Vector{<:NamedTuple}.
• date keywords of AbstractArray will return a Vector{String} or Vector{<:NamedTuple}.
• date keywords of Tuple{start,end} will take all the dates between the start and end dates as a Vector{String} or Vector{<:NamedTuple}.

Where date and month keywords coexist, Vector{Vector{String}} of Vector{Vector{NamedTuple}} is the result. date ranges are always the outer Vector, month the inner Vector with layer tuples as the inner NamedTuple. No other keywords can be Vector.

RasterDataSource

Abstract supertype for raster data collections.

ALWB{Union{Deciles,Values},Union{Day,Month,Year}} <: RasterDataSource

Data from the Australian Landscape Water Balance (ALWB) data source.

See: www.bom.gov.au/water/landscape

The dataset contains NetCDF files. They have a time dimension so that multiple dates are stored in each file.

The available layers are: (:rain_day, :s0_pct, :ss_pct, :sd_pct, :sm_pct, :qtot, :etot, :e0, :ma_wet, :pen_pet, :fao_pet, :asce_pet, :msl_wet, :dd), available in daily, monthly and annual resolutions, and as Values or relative Deciles.

getraster for ALWB must use a date keyword to specify the date to download.

Usage with getraster

getraster(source::Type{<:ALWB{Union{Deciles,Values},Union{Day,Month,Year}}}, [layer]; date)

Download ALWB weather data as values or deciles with timesteps of Day, Month or Year.

Arguments

• layer: Symbol or Tuple of Symbol from (:rain_day, :s0_pct, :ss_pct, :sd_pct, :sm_pct, :qtot, :etot, :e0, :ma_wet, :pen_pet, :fao_pet, :asce_pet, :msl_wet, :dd). Without a layer argument, all layers will be downloaded, and a NamedTuple of paths returned.

Keywords

• date: a DateTime, AbstractVector of DateTime or a Tuple of start and end dates. For multiple dates, a Vector of multiple filenames will be returned. ALWB is available with a daily, monthly, and yearly, timestep.

Example

This will return the file containing annual averages, including your date:

julia> getraster(ALWB{Values,Year}, :ss_pct; date=Date(2001, 2))
"/your/RASTERDATASOURCES_PATH/ALWB/values/month/ss_pct.nc"

Returns the filepath/s of the downloaded or pre-existing files.

AWAP <: RasterDataSource

Daily weather data from the Australian Water Availability Project, developed by CSIRO.

See: www.csiro.au/awap

The AWAP dataset contains ASCII .grid files.

Usage with getraster

getraster(source::Type{AWAP}, [layer]; date)

Arguments

• layer: Symbol or Tuple of Symbol for layers in (:solar, :rainfall, :vprpress09, :vprpress15, :tmin, :tmax). Without a layer argument, all layers will be downloaded, and a NamedTuple of paths returned.

Keywords

• date: a DateTime, AbstractVector of DateTime or a Tuple of start and end dates. For multiple dates, A Vector of multiple filenames will be returned. AWAP is available with a daily timestep.

Example

Download rainfall for the first month of 2001:

julia> getraster(AWAP, :rainfall; date=Date(2001, 1, 1):Day(1):Date(2001, 1, 31))

31-element Vector{String}:
 "/your/path/AWAP/rainfall/totals/20010101.grid"
 "/your/path/AWAP/rainfall/totals/20010102.grid"
 ...
 "/your/path/AWAP/rainfall/totals/20010131.grid"

Returns the filepath/s of the downloaded or pre-existing files.

CHELSA{Union{BioClim,BioClimPlus,Climate,<:Future}} <: RasterDataSource

Data from CHELSA, currently implements the BioClim, BioClimPlus, and Climate variables for current and future conditions.

See: chelsa-climate.org for the dataset, and the getraster docs for implementation details.

Usage with getraster

getraster(source::Type{CHELSA{BioClim}}, [layer]; [version], [patch])
getraster(source::Type{CHELSA{BioClimPlus}}, [layer]; [version], [patch])
getraster(T::Type{CHELSA{Climate}}, [layer]; month)
getraster(T::Type{CHELSA{Future{BioClim}}}, [layer]; date
getraster(T::Type{CHELSA{Future{Climate}}}, [layer]; date, month)

Download CHELSA BioClim data from chelsa-climate.org.

Arguments

• layer: Integer, Symbol or tuple/range of these. Without a layer argument, all layers will be downloaded, and a NamedTuple of paths returned. Available layers are:
  • BioClim: Integers 1 to 19 or Symbols :bio1 to :bio19
  • BioClimPlus: Includes BioClim layers and many additional layers. See RasterDataSources.layers(BioClimPlus).
  • Climate: (:clt, :cmi, :hurs, :ncdf, :pet, :pr, :rsds, :sfcWind, :tas, :tasmax, :tasmin, :vpd)

Keyword arguments

• version: Integer indicating the CHELSA version, currently either 1 or 2. Defaults to 2.
• patch: Integer indicating the CHELSA patch number. Defaults to the latest patch (V1.2 and V2.1)
• month: Integer or AbstractArray of Integer. Chosen from 1:12.
• date: a Date or DateTime object, a Vector, or Tuple of start/end dates. Note that CHELSA CMIP5 only has two datasets, for the periods 2041-2060 and 2061-2080. CMIP6 has datasets for the periods 2011-2040, 2041-2070, and 2071-2100. Dates must fall within these ranges.

Example

using RasterDataSources, Dates
getraster(CHELSA{Future{BioClim, CMIP6, GFDL_ESM4, SSP370}}, 1, date = Date(2050))

EarthEnv{Union{HabitatHeterogeneity,LandCover}} <: RasterDataSource

Data from the EarthEnv including HabitatHeterogeneity and LandCover

See: www.earthenv.org

Usage with getraster

getraster(T::Type{EarthEnv{HabitatHeterogeneity}}, [layer]; res="25km")
getraster(T::Type{EarthEnv{LandCover}}, [layer]; discover=false)

Download EarthEnv habitat heterogeneity data.

Arguments

• layer: Integer, Symbol or tuple/range of these. Without a layer argument, all layers will be downloaded, and a NamedTuple of paths returned. Available layers are:
  • HabitatHeterogeneity: (:cv, :evenness, :range, :shannon, :simpson, :std, :Contrast, :Correlation, :Dissimilarity, :Entropy, :Homogeneity, :Maximum, :Uniformity, :Variance)
  • LandCover: Integers 1 to 12 or Symbols (:needleleaftrees, :evergreenbroadleaftrees, :deciduousbroadleaftrees, :othertrees, :shrubs, :herbaceous, :cultivatedandmanaged, :regularlyflooded, :urbanbuiltup, :snowice, :barren, :openwater)

Keywords

• res: String chosen from ("1km", "5km", "25km"), defaulting to "25km".
• discover::Bool: whether to download the dataset that integrates the DISCover model.

Returns the filepath/s of the downloaded or pre-existing files.

WorldClim{Union{BioClim,Climate,Elevation,Weather,<:Future}} <: RasterDataSource

Data from WorldClim datasets, either BioClim, Climate, Weather, Climate, or Future variables for current and future conditions.

Future variables are available for BioClim and Climate data. See the docstring of Future for available model choices and implementation details.

See: www.worldclim.org

Usage with getraster

getraster(T::Type{WorldClim{BioClim}}, [layer::Union{Tuple,Int,Symbol}]; res)
getraster(T::Type{WorldClim{Climate}}, [layer::Union{Tuple,Symbol}]; month, res)
getraster(T::Type{WorldClim{Elevation}}; res)
getraster(T::Type{WorldClim{Weather}}, [layer::Union{Tuple,Symbol}]; date)
getraster(T::Type{WorldClim{Future}}, [layer]; date, res) => String

Arguments

• layer: Integer, Symbol or tuple/range of these. Without a layer argument, all layers will be downloaded, and a NamedTuple of paths returned. Available layers are:
  • BioClim: Integers 1 to 19 or Symbols :bio1 to :bio19
  • Climate: (:tmin, :tmax, :tavg, :prec, :srad, :wind, :vapr)
  • Elevation: Only has a single layer, :elev
  • Weather: (:tmin, :tmax, :prec)
  • Future{Climate}: (:tmin, :tmax, :prec)

Keywords

• res: String chosen from ("30s", "2.5m", "5m", "10m"), "10m" by default.
• month: Integer or AbstractArray of Integer. Chosen from 1:12.
• date: a Date or DateTime object, a Vector of dates, or Tuple of start/end dates. WorldClim{Weather} is available with a daily timestep. Future data is available in 20-year intervals from 2021 to 2100.

Examples

using RasterDataSources, Dates
bio_current = getraster(WorldClim{BioClim}, res = "5m")
bio_future = getraster(WorldClim{Future{BioClim, CMIP6, GFDL_ESM4, SSP370}}, date = Date(2050), res = "5m")

MODIS{ModisProduct} <: RasterDataSource

MODIS/VIIRS Land Product Database. Vegetation indices, surface reflectance, and more land cover data. Data from ModisProducts datasets.

See: modis.ornl.gov

Usage with getraster

getraster(T::Union{Type{<:ModisProduct}, Type{MODIS{X}}}, [layer::Union{Tuple,AbstractVector,Integer, Symbol}]; kwargs...) => Union{String, AbstractVector, NamedTuple}

Download MODIS data for a given ModisProduct as ASCII raster(s).

Arguments

• layer: Integer or tuple/range of Integer or Symbols. Without a layer argument, all layers will be downloaded, and a NamedTuple of paths returned.

Available layers for a given product can be looked up using RasterDataSources.layerkeys(T::Type{<:ModisProduct}).

Keywords

• lat and lon: Coordinates in decimal degrees of the approximate center of the raster. The MODIS API will try to match its pixel grid system as close as possible to those coordinates.

• km_ab and km_lr: Half-width and half-height of the raster in kilometers (kilometers above/below and left/right). Currently only Integer values are supported, up to 100.

• date: String, Date, DateTime, AbstractVector of dates or Tuple of a start and end date for the request. Strings should be in format YYYY-MM-DD but can be in similar formats as long as they are comprehensible by Dates.Date. The available date interval for MODIS is 16 days, reset every first of January.

Example

Download 250m NDVI in the western part of Britanny, France, from winter to late spring, 2002:

julia> getraster(MOD13Q1, :NDVI; lat = 48.25, lon = -4, km_ab = 50, km_lr = 50, date = (Date(2002,1,1), Date(2002,6,1)))

    10-element Vector{String}:
    "/your/path/MODIS/MOD13Q1/250m_16_days_NDVI/47.8313_-4.5899_2002-01-01.asc"
    ...
    "/your/path/MODIS/MOD13Q1/250m_16_days_NDVI/47.8313_-4.5899_2002-05-25.asc"

Will attempt to download several files, one for each date and layer combination, and returns the filepath/s of the downloaded or pre-existing files. Coordinates in the file names correspond to the lower-left corner of the raster.

ModisProduct <: RasterDataSet

Abstract supertype for MODIS/VIIRS products.

Usage

Some commonly used products are MOD13Q1 (250m resolution MODIS vegetation indices) and VNP13A1 (500m resolution VIIRS vegetation indices). Refer to the MODIS documentation for detailed product information.

layerkeys(T::Type{<:ModisProduct}) => Tuple

Tuple of Symbols corresponding to the available layers for a given product. May issue a request to MODIS server to get the layers list, or might just read this information if the correctly named file is available.

RasterDataSet

Abstract supertype for datasets that belong to a RasterDataSource.

BioClim <: RasterDataSet

BioClim datasets. Usually containing layers from 1:19. These can also be accessed with :bioX, e.g. :bio5.

They do not usually use month or date keywords, but may use date in past/future scenarios.

Currently implemented for WorldClim and CHELSA as WorldClim{BioClim}, WorldClim{Future{BioClim}}, CHELSA{BioClim}, CHELSA{Future{BioClim, args..}}.

See the CHELSA and WorldClim docs for implementation details.

BioClimPlus <: RasterDataSet

Extended BioClim datasets, available from CHELSA. More information on the CHELSA website: https://chelsa-climate.org/exchelsa-extended-bioclim/

Some of these are available as average annual maximum, minimum, mean, and range. Others have a single value, more like the regular BioClim variables.

They do not usually use month or date keywords, but may use date in past/future scenarios.

Currently implemented for CHELSA as CHELSA{BioClim} and CHELSA{Future{BioClim, args..}}, specifying layer names as Symbols.

See the CHELSA docs for implementation details.

Climate <: RasterDataSet

Climate datasets. These are usually months of the year, not specific dates, and use a month keyword in getraster. They also use date in past/future scenarios.

Currently implemented for WorldClim and CHELSA as WorldClim{Climate}, WorldClim{Future{Climate, args..}} CHELSA{Climate} and CHELSA{Future{Climate, args..}}.

See the CHELSA and WorldClim docs for implementation details.

Weather <: RasterDataSet

Weather datasets. These are usually large time-series of specific dates, and use a date keyword in getraster.

Currently implemented for WorldClim as WorldClim{Weather}.

See the WorldClim docs for implementation details.

Elevation <: RasterDataSet

Elevation datasets.

Currently implemented for WorldClim as WorldClim{Elevation}.

See the WorldClim docs for implementation details.

LandCover <: RasterDataSet

Land-cover datasets.

Currently implemented for EarthEnv as EarthEnv{LandCover}.

See the EarthEnv docs for implementation details.

HabitatHeterogeneity <: RasterDataSet

Habitat heterogeneity datasets.

Currently implemented for EarthEnv as EarthEnv{HabitatHeterogeneity}.

See the EarthEnv docs for implementation details.

Future{<:RasterDataSet,<:CMIPphase,<:ClimateModel,<:ClimateScenario}

Future climate datasets specified with a dataset, phase, model, and scenario.

Type Parameters

RasterDataSet

Currently BioClim and Climate are implemented for CHELSA and WorldClim. Future WorldClim is only available for CMIP6.

CMIPphase

Can be either CMIP5 or CMIP6.

ClimateModel

Climate models can be chosen from:

RasterDataSources.ACCESS1_0, RasterDataSources.BNU_ESM, RasterDataSources.CCSM4, RasterDataSources.CESM1_BGC, RasterDataSources.CESM1_CAM5, RasterDataSources.CMCC_CMS, RasterDataSources.CMCC_CM, RasterDataSources.CNRM_CM5, RasterDataSources.CSIRO_Mk3, RasterDataSources.CanESM2, RasterDataSources.FGOALS_g2, RasterDataSources.FIO_ESM, RasterDataSources.GFDL_CM3, RasterDataSources.GFDL_ESM2G, RasterDataSources.GFDL_ESM2M, RasterDataSources.GISS_E2_H_CC, RasterDataSources.GISS_E2_H, RasterDataSources.GISS_E2_R_CC, RasterDataSources.GISS_E2_R, RasterDataSources.HadGEM2_AO, RasterDataSources.HadGEM2_CC, RasterDataSources.IPSL_CM5A_LR, RasterDataSources.IPSL_CM5A_MR, RasterDataSources.MIROC_ESM_CHEM, RasterDataSources.MIROC_ESM, RasterDataSources.MIROC5, RasterDataSources.MPI_ESM_LR, RasterDataSources.MPI_ESM_MR, RasterDataSources.MRI_CGCM3, RasterDataSources.MRI_ESM1, RasterDataSources.NorESM1_M, RasterDataSources.bcc_csm_1, RasterDataSources.inmcm4 for CMIP5;

RasterDataSources.ACCESS_CM2, RasterDataSources.BCC_CSM2_MR, RasterDataSources.CMCC_ESM2, RasterDataSources.EC_Earth3_Veg, RasterDataSources.FIO_ESM_2_0, RasterDataSources.GFDL_ESM4, RasterDataSources.GISS_E2_1_G, RasterDataSources.HadGEM3_GC31_LL, RasterDataSources.INM_CM5_0, RasterDataSources.IPSL_CM6A_LR, RasterDataSources.MIROC6, RasterDataSources.MPI_ESM1_2_HR, RasterDataSources.MRI_ESM2_0, RasterDataSources.UKESM1_0_LL for CMIP6;"

ClimateScenario

CMIP5 Climate scenarios are all RepresentativeConcentrationPathway and can be chosen from: RCP26, RCP45, RCP60, RCP85

CMIP6 Climate scenarios are all SharedSocioeconomicPathway and can be chosen from: SSP126, SSP245, SSP370, SSP585

However, note that not all climate scenarios are available for all models.

Example

using RasterDataSources
datasource = CHELSA{Future{BioClim, CMIP5, GFDL_ESM4, SSP370}}

ClimateModel

Abstract supertype for climate models use in Future datasets.

CMIPphase

Abstract supertype for phases of the CMIP, the Coupled Model Intercomparison Project.

Subtypes are CMIP5 and CMIP6.

CMIP5 <: CMIPphase

The Coupled Model Intercomparison Project, Phase 5.

CMIP6 <: CMIPphase

The Coupled Model Intercomparison Project, Phase 6.

ClimateScenario

Abstract supertype for scenarios used in CMIPphase models.

RepresentativeConcentrationPathway

Abstract supertype for Representative Concentration Pathways (RCPs) for CMIP5.

Subtypes are: RCP26, RCP45, RCP60, RCP85

SharedSocioeconomicPathway

Abstract supertype for Shared Socio-economic Pathways (SSPs) for CMIP6.

Subtypes are: SSP126, SSP245, SSP370, SSP585

Values <: DataMode

Get the dataset as regular measured values.

Deciles <: DataMode

Get the dataset in relative deciles.

layerkeys(T::Type{<:ModisProduct}) => Tuple

Tuple of Symbols corresponding to the available layers for a given product. May issue a request to MODIS server to get the layers list, or might just read this information if the correctly named file is available.

rastername(source::Type, [layer]; kw...)

Returns the name of the file, without downloading it.

Arguments are the same as for getraster

Returns a String or multiple Strings.

rasterpath(source::Type, [layer]; kw...)

Returns the name of the file, without downloading it.

Arguments are the same as for getraster

Returns a String or multiple Strings.

rasterurl(source::Type, [layer]; kw...)

If the file has a single url, returns it without downloading.

Arguments are the same as for getraster.

Returns a URIs.jl URI or mulitiple URIs.

zipname(source::Type, [layer]; kw...)

If the url is a zipped file, returns its name.

Arguments are as the same for getraster where possible.

Returns a String or multiple Strings.

zippath(source::Type, [layer]; kw...)

If the url is a zipped file, returns its path when downloaded. (This may not exist after extraction with getraster)

Arguments are the same as for getraster where possible.

Returns a String or multiple Strings.

zipurl(source::Type, [layer]; kw...)

If the url is a zipped file, returns its zip path without downloading.

Arguments are the same as for getraster where possible.

Returns a URIs.jl URI or mulitiple URIs.

modis_request(T::Type{<:ModisProduct}, args...)

Lowest level function for requests to modis server. All arguments are assumed correct.

Arguments

• layer: String matching the "exact" layer name (i.e. as it is written in the MODIS dataset itself) for the given product. e.g. "250m_16_days_EVI".

• lat, lon, km_ab, km_lr in correct types

• from, to: Strings of astronomical dates for start and end dates of downloaded data, e.g. "A2002033" for "2002-02-02"

Returns a NamedTuple of information relevant to build a raster header, and a Vector of Dicts containing raster data, directly downloaded from MODIS. Those will almost always directly be passed to RasterDataSources.process_subset

process_subset(T::Type{<:ModisProduct}, subset::Vector{Any}, pars::NamedTuple)

Process a raw subset and argument parameters and create several raster files. Any already existing file is not overwritten.

For each band, a separate folder is created, containing a file for each of the required dates. This is inspired by the way WorldClim{Climate} treats the problem of possibly having to download several dates AND bands.

Can theoretically be used for MODIS data that does not directly come from RasterDataSources.modis_request, but caution is advised.

Returns the filepath/s of the created or pre-existing files.

product(T::Type{<:ModisProduct})

Extracts ModisProduct product name as a String

sinusoidal_to_latlon(x::Real, y::Real)

Convert x and y in sinusoidal projection to lat and lon in dec. degrees

This function uses Proj for coordinate conversions. Use import Proj to make it available.