PADRE - The PetaFLOP AARTFAAC Data-Reduction Engine

The PADRE project aimed to accelerate and improve theprocessing pipeline of the AARTFAAC telescope. AARTFAAC is a LOFAR derivative, that piggybacks on individual LOFAR Low-Band Antennas. It creates an all-sky image every second in real time, in order to detect transients on the sky. The aim of this project was to reduce the latency of the whole processing pipeline from 1.5 days to 7 seconds, and to make the pipeline scalable to a higher number of antennas and more bandwidth, so that the image quality would be improved. The upgrade of the LOFAR station hardware and network will allow much higher data rates than what was previously possible with AARTFAAC.
The full processing pipeline consists of four components: the station processing on FPGAs (not covered by this project), the correlator that combines the antenna data, the calibration/imaging pipeline, and the transient detection pipeline. To achieve the aforementioned goals, we had to accelerate and improve many components throughout the last three parts of the processing pipeline.

We achieved the goals of this project partially. The correlator has become two orders of magnitude faster than the 9-year old GPU correlator, thanks to the use of new tensor-core GPU technology. The imaging/calibration pipeline underwent even larger changes: as the original imager did not calibrate the extended, irregular antenna field beyond the compact LOFAR superterp well, we replaced the imaging pipeline by the pipeline that is used for regular LOFAR observations, and started making improvements there. Some of these improvements were AARTFAAC specific, and others benefit regular LOFAR observations as well. Although many improvements were made, the deconvolution step still takes too long (tens of seconds), but we can work around this by doing the expensive computations only on part of the data. The transient-detection part of the pipeline, that previously did not run in real time at all, has been greatly modernized and optimized.
Unfortunately, the upgrade of the LOFAR stations has been delayed, and will not be completed before late 2025, so it was impossible to demonstrate the enhanced pipeline in real time. To circumvent this, we used raw antenna data that was captured by the LOFAR Transient Buffer Boards (TBB), and developed software that converts this data to data that looks like antenna data to the AARTFAAC correlator. This way, we could simulate what would be possible after the LOFAR upgrade, albeit not in real time and only for four seconds of data; enough to create four all-sky images, but not enough to discover transients. The image result is shown in Figure 1. Obviously, the proof of the pudding is in the eating, and we look forward to real LOFAR 2.0 antenna data.

This pages covers two projects, the eTEC-BIG project PADRE and the software sustainability project PySE.