The ambition of the MINERVA project is to establish a network that facilitates and strengthens cooperation between upstream and downstream users to bolster the Dutch position in agricultural vegetation monitoring using microwaves. The Netherlands eScience Center participated in Work Package WP240 of this project, which focused on developing an advanced data pipeline for processing microwave remote sensing data. In collaboration with researchers from Delft University of Technology (TUDelft) and Wageningen University & Research (WUR), the eScience Center developed RS-DAT (Remote Sensing Deployable Analysis environmenT), which addresses the advanced data processing needs of MINERVA stakeholders.
RS-DAT is a generic framework for Earth Observation (EO) data processing, providing a suite of data analysis tools for various EO applications. It allows users to create an environment for exploring and analyzing remote sensing data on high-performance and high-throughput computing (HPC/HTC) systems, along with associated storage resources. Built on the Pangeo ecosystem, RS-DAT seamlessly integrates and extends the Python and PyData ecosystems—including Dask and Xarray—and offers tools for managing data storage and retrieval on mass storage systems (e.g., dCache at SURF).
Through the developments in the MINERVA project, the following key needs of the Food and Agriculture discipline have been addressed:
- Developing state-of-the-art models for Food and Agriculture
Numerical models, such as Land-Surface Models (LSMs) and Crop Simulation Models (CSMs), are crucial for understanding the physical processes involved in agriculture. These models depend on data-intensive, computationally demanding techniques such as Machine Learning (ML) and Data Assimilation (DA), which require high-performance computing infrastructure. RS-DAT provides researchers with seamless access to computational resources and user-friendly data tools, enabling the efficient development of such models. - Monitoring large-scale, real-time events
Anomalous climate events—such as droughts, wildfires, and floods—can significantly impact agricultural activities. High-resolution satellite data, both spatial and temporal, offers new opportunities for real-time monitoring of these events. The RS-DAT framework simplifies access to open satellite data repositories and integrates data acquisition into research workflows. Additionally, it connects with large-scale storage systems, such as SURF's dCache, to handle extensive datasets efficiently. - Deploying novel algorithms for local challenges
RS-DAT enables researchers to streamline the deployment of their algorithms on local infrastructures, allowing global stakeholders to efficiently apply peer-reviewed, customized data solutions. The framework supports various computational platforms, including SLURM-managed HPC systems and cloud infrastructures, enabling the scalable deployment of software and workflows across available resources.
