Provenance: what a result was built from

A simulation is built from published data - climate layers, land cover, a region's outline, occurrence records - and from the software that ran it. provenance answers, for a study area, a habitat or an ecosystem, what that was: the package version and the DOI it is cited by, the grid, the run, and a record of every input. write_provenance writes the same answer as a TOML file, which is the file to commit beside a paper's figures so that a reader can see what made them.

What is recorded

Every file the package fetches gets a record written beside it - where it came from, when, its size and checksum, and the dataset's DOI, licence, version and citation from the catalogue (see Layers). When a study area reads a layer, it keeps a record of each file it read, and those records stay with the area, and with a habitat or an ecosystem built on it, after the reads themselves are discarded. The files of a shape or mask that cut the grid are recorded the same way. A file with no record of its own - one you handed over rather than one the package fetched - is recorded by its name, with the catalogue's facts about its dataset where it is a layer of one.

Each input is an InputRecord, and a record never holds an absolute path, so the file can be shared as it stands.

Recording data the package did not fetch

Occurrence records, trait tables and parameters from the literature reach a model through your own code, so the package cannot know where they came from. Say so with the provenance keyword, which takes an InputRecord or a vector of them: on build_species for the species, on build_ecosystem for what belongs to the run as a whole, and on an Intervention for data an intervention is built from - whose records join the ecosystem's once it has acted.

using EcoSISTEM

seeding = EcoSISTEM.InputRecord(role = :abundance,
                                dataset = "GBIF occurrence download",
                                doi = "10.15468/dl.abc123",
                                licence = "CC BY-NC 4.0")
species = build_species(DefaultEcosystem(), numspecies = 3, verbosity = :silent)
habitat = build_habitat(DefaultEcosystem(), verbosity = :silent)
eco = build_ecosystem(species, habitat, seed = 1, provenance = seeding)
provenance(eco)

Provenance
  software  EcoSISTEM 0.8.0 (doi 10.5281/zenodo.4705053), Julia 1.13.0
  grid      100 of 100 cells active, cells of 1.0 km, synthetic
  run       seed 1, 0.0 s elapsed
  inputs
    GBIF occurrence download (doi 10.15468/dl.abc123)
      abundance

This ecosystem is built on a synthetic grid, so it read no files: its only input is the record given to it.

Writing it out

path = write_provenance(joinpath(mktempdir(), "provenance.toml"), eco)
print(read(path, String))

[grid]
active = 100
cells = 100
cellsize = "1.0 km"

    [grid.extent]
    X = ["0.0 km", "10.0 km"]
    Y = ["0.0 km", "10.0 km"]

[[inputs]]
dataset = "GBIF occurrence download"
doi = "10.15468/dl.abc123"
licence = "CC BY-NC 4.0"
role = "abundance"

[run]
calendar = "ExactDates()"
elapsed = "0.0 s"
seed = 1

[software]
doi = "10.5281/zenodo.4705053"
julia = "1.13.0"
package = "EcoSISTEM"
version = "0.8.0"

Blank fields are left out. Quantities, the coordinate reference system and the grid's extent are written as text, and fetch times as TOML dates.

What a record does not cover

A raster handed around on its own - one read with read(spec) and passed to a function - carries no record; ask the spec or the study area it came from. A recorder - RecordAbundance, RecordDiversity, SaveAbundance - keeps the run's provenance as it stood at its last write, so write_provenance(path, recorder) writes it; SaveAbundance writes it beside every file it saves. An array you fill yourself from a callback carries none, so write the ecosystem's provenance beside it.