DIRAC

Distributed radio astronomical computing

Image by: Adolf Schaller NASA-MSFC

The infant Universe (its first-billion years) remains its least explored era. Although sparse observations are available, the red-shifted 21-cm emission of neutral hydrogen (HI) –seen as spectral fluctuations at several meter wavelengths– allows this era to be opened up for much more detailed study. The HI signal, however, is orders of magnitude fainter than most contaminating signals (e.g. (extra)Galactic foregrounds). Nonetheless, it is possible to detect and study this weak HI signal using the latest generation low-frequency radio interferometers (e.g. LOFAR, MWA),provided that all systematic (instrumental, ionospheric, etc.) errors are eliminated to sufficient levels (i.e. “calibrated”). These errors are determined and removed by solving a complex non-linear optimization problem with millions of unknown parameters, constrained by many terabytes of data.

Data parallelism is inherently exploited in calibration of radio-interferometric observations, where calibration is done in parallel on data at different frequencies. However, to achieve the highest accuracy and precision in calibration, without biasing the weak HI signal, a global calibration scheme is needed. We have demonstrated that this can be done using consensus optimization. In this project we will develop this from a proof-of-concept to a fully capable, computationally efficient and scalable software system. Ensuing orders-of-magnitude improvements in accuracy and computational speed not only enable the detection of this weak HI signal but also benefit a wider astronomical community (those using e.g. LOFAR, MWA, MeerKAT, ASKAP, APERTIF, SKA). The software developed in this project will be made publicly available for many other distributed optimization applications.

Participating organisations

ASTRON
Netherlands eScience Center
University of Groningen
Natural Sciences & Engineering
Natural Sciences & Engineering

Impact

Output

Team

SY
Sarod Yatawatta
Ben van Werkhoven
Ben van Werkhoven
Senior eScience Research Engineer
Netherlands eScience Center
AM
Adriënne Mendrik
eScience Coordinator
Netherlands eScience Center
Faruk Diblen
Faruk Diblen
eScience Research Engineer
Netherlands eScience Center
Hanno Spreeuw
Hanno Spreeuw
eScience Research Engineer
Netherlands eScience Center
Rena Bakhshi
Rena Bakhshi
Coordinator
Netherlands eScience Center
LK
Leon Koopmans
Co Applicant
Rijksuniversiteit Groningen

Related projects

PADRE - The PetaFLOP AARTFAAC Data-Reduction Engine

Improving the AARTFAAC processing pipeline

Updated 6 months ago
Finished

EOSCpilot LOFAR

Unlocking the LOFAR Long Term Archive

Updated 18 months ago
Finished

AA-ALERT

Access and acceleration of the Apertif Legacy exploration of the radio transient sky

Updated 22 months ago
Finished

Compressing the sky into a large collection of statistical models

Optimized data handling for observations in astronomy

Updated 18 months ago
Finished

AMUSE

The evolution of embedded star clusters

Updated 18 months ago
Finished

Beyond the Data Explosion (eAstronomy)

An eScience infrastructure for huge interferometric datasets

Updated 18 months ago
Finished

Related software

Kernel Tuner

KE

Kernel Tuner greatly simplifies the development of highly-optimized and auto-tuned CUDA, OpenCL, and C code, supporting many advanced use-cases and optimization strategies that speed up the auto-tuning process.

Updated 12 months ago
105 15

Sagecal

SA

Fast, memory efficient and GPU accelerated radio interferometric calibration program

Updated 4 days ago
102 4