ESMValCore

ESMValTool Core: core functionalities and driver for ESMValTool, a community diagnostic and performance metrics tool for routine evaluation of Earth System Models in CMIP.

35
mentions
41
contributors

Cite this software

DOI:

10.5281/zenodo.3387139

What ESMValCore can do for you

  • Finding data in a directory structure typically used for CMIP data
  • ESMValTool preprocessor functions based on iris for e.g. regridding, vertical interpolation, statistics, correcting (meta)data errors, extracting a time range, etcetera.
  • High flexibility: new diagnostics and more observational data can be easily added.
  • Multi-language support: Python, NCL, R, Julia... other open-source languages are possible.
  • CF/CMOR compliant: data from many different projects can be handled (CMIP, obs4mips, ana4mips, CCMI, CCMVal, AEROCOM, etc.). Routines are provided to CMOR-ize non-compliant data.
  • Integration in modeling workflows: for EMAC, NOAA-GFDL and NEMO, can be easily extended.

The ESMValCore software package provides the core functionality for ESMValTool, the Earth System Model eValuation Tool. It provides a configurable framework for finding CMIP files using a “data reference syntax”, applying commonly used pre-processing functions to them, running analysis scripts, and recording provenance. Numerous pre-processing functions, e.g. for data selection, regridding, and statistics are readily available and the modular design makes it easy to add more. The ESMValCore package is easy to install with relatively few dependencies, written in Python 3, based on state-of-the-art open-source libraries such as Iris and Dask, and widely used standards such as YAML, NetCDF, CF-Conventions, and W3C PROV. An extensive set of automated tests and code quality checks ensure the reliability of the package.

The ESMValCore package uses human-readable recipes to define which variables and datasets to use, how to pre-process that data, and what scientific analysis scripts to run. The package provides convenient interfaces, based on the YAML and NetCDF/CF-convention file formats, for running diagnostic scripts written in any programming language. Because the ESMValCore framework takes care of running the workflow defined in the recipe in parallel, most analyses run much faster, with no additional programming effort required from the authors of the analysis scripts. For example, benchmarks show a factor of 30 speedup with respect to version 1 of the tool for a representative recipe on a 24 core machine. A large collection of standard recipes and associated analysis scripts is available in the ESMValTool package for reproducing selected peer-reviewed analyses.

Keywords
  • Big data
  • Optimized data handling
  • Visualization
  • Workflow technologies
Programming language
  • Python 97%
  • HTML 3%
License
  • Apache-2.0
</>Source code

Participating organisations

Alfred Wegener Institute
Barcelona Supercomputing Center
DLR
ETH Zurich
Ludwig Maximilian University of Munich
Met Office
Netherlands eScience Center
Plymouth Marine Laboratory
Swedish Meteorological and Hydrological Institute
University of Bremen
University of Reading

Mentions

Contributors

Contact person

Bouwe Andela

Bouwe Andela

Netherlands eScience Center
Mail Bouwe
ASS
Abel Soares Siqueira
AS
Alistair Sellar
Met Office
AL
Axel Lauer
DLR
BV
Barbara Vreede
BM
Benjamin Müller
Ludwig Maximilian University of Munich
BG
Bettina Gier
University of Bremen
BL
Bill Little
MetOffice, UK
BH
Birgit Hassler
DLR
BB
Bjoern Broetz
DLR, Germany
BB
Björn Brötz
DLR
Bouwe Andela
Bouwe Andela
EG
Evgenia Galytska
FA
Fakhereh Alidoost
Netherlands eScience Center
Faruk Diblen
Faruk Diblen
Netherlands eScience Center
Inti Pelupessy
Inti Pelupessy
Netherlands eScience Center
Jaro Camphuijsen
Jaro Camphuijsen
Netherlands eScience Center
JV
Javier Vegas-Regidor
Barcelona Supercomputing Center
KW
Katja Weigel
KZ
Klaus Zimmermann
Swedish Meteorological and Hydrological Institute
LD
Laura Dreyer
MetOffice, UK
LdM
Lee de Mora
Plymouth Marine Laboratory
MS
Manuel Schlund
DLR
MJ
Martin Jury
MH
Mathias Hauser
Institute for Atmospheric and Climate Science, ETH Zurich, Zurich, Switzerland
MR
Mattia Righi
DLR
Niels  Drost
Niels Drost
Netherlands eScience Center
NK
Nikolay Koldunov
Alfred Wegener Institute
PE
Paul Earnshaw
MetOffice, UK
PC
Pep Cos-Espuña
BSC, Spain
Peter Kalverla
Peter Kalverla
Netherlands eScience Center
RL
Ruth Lorenz
ETH Zurich
RK
Rémi Kazeroni
DLR
SL
Saskia Loosveldt-Tomas
Barcelona Supercomputing Center
Stef Smeets
Stef Smeets
Netherlands eScience Center
SS
Stéphane Sénési
Stéphane Sénési EIRL, Colomiers, France
SK
Sujan Koirala
Max Planck Institute for Biogeochemistry, Germany
TS
Tobias Stacke
Helmholtz-Zentrum Geesthacht, Germany
VP
Valeriu Predoi
University of Reading
VE
Veronika Eyring
University of Bremen, DLR

Related projects

IS-ENES3

Providing the infrastructure to better understand and project climate variability and change

Updated 2 months ago
Running

eWaterCycle II

Overcoming the challenge of locality using a community multi-model environment

Updated 2 months ago
Running

EUropean Climate Prediction system (EUCP)

An innovative European regional ensemble climate prediction system

Updated 2 months ago
Finished

MAGIC

Metrics and Access to Global Indices for Climate Projections

Updated 2 months ago
Finished

Related tools

ESMValTool

ES

The Earth System Model eValuation Tool is a community diagnostics and performance metrics tool for the evaluation of Earth System Models that allows for routine comparison of models and observations.

Updated 2 months ago
80 mentions, 84 contributors