HiPeRTA is a C++ library implementing the real-time analysis low-level reconstruction algorithms of the Cherenkov Telescope Array (CTA). The real-time analysis is used by the Science Alert Generation (SAG) subsystem of the Array Control And Data Acquisition (ACADA) of CTAO for data quality and science monitoring and science alert generation.

HiPeRTA implements the reconstruction algorithms to produce DL1 level data from lower level data. The library supports R0, R1 or DL0 data as input, in the form of hdf5 files or streamed hdf5 or protobuf data, and outputs DL1 hdf5 files compliant with CTAO format.

To achieve the best reconstruction performances while running in real time, HiPeRTA uses a custom high-performance computing back-end taking advantage of the latest SIMD operations available to modern processors, for native vectorization.

Code

The code is available in 2 (mirrors) repositories:

• CTAO ACADA repository
• LAPP repository

Documentation

For Users: HiPeRTA entrypoints and configuration file

Wiki: Real-time analysis wiki

Developper: HiPeRTA API

Dependencies

Build only dependencies

HiPeRTA's core is a computing library implementing the processing algorithms to process data up to the DL1 data level. It is provided without any other dependency to simplify its integration in other systems. It is written in C++ and requires a recent compiler to get the best performances:

• c++ compiler (g++>9.2 or clang>10.0)
• cmake >= 3.0
• make >= 4.0

I/O dependencies

HiPeRTA can also be used a stand alone processing library for CTAO data. The following dependencies are then required to manage the data input and output:

• hdf5 (reading/writing data files)
• blosc (compression library for hdf5 files)
• zeromq (messaging interface to receive data streamed over the network)

installation example

Users can install the dependencies as they prefer, as long as they are on $LD_LIBRARY_PATH (or $PATH for cmake, make and g++).

Ubuntu

sudo apt-get install g++ cmake make libhdf5-dev libhdf5-cpp-103  libblosc-dev libblosc1 libzmq3-dev

Conda

A common usage is to install the dependencies using the conda package manager. The following command will create an environment named hiperta with the dependencies installed:

conda create -n hiperta -c conda-forge "cmake>=3" "make>=4" "gcc_linux-64>=7.5" "gxx_linux-64>=7.5" "hdf5" "blosc" "cppzmq" "sysroot_linux-64=2.17"

Notes:

• The compilers installed with conda (gcc_linux-64) don't bring the correct sysroot file, so the correct one needs to be explicitely installed.

Dependencies from the ESCAPE project

These dependencies are automatically downloaded by cmake from the in2p3 gitlab if they are not already installed in the environment.

• DataStream: load/write binary data read from file or streamed over network.
• StringUtils: String manipulation in C++.
• PhoenixThread: Utilities to simplify the use of threads in C++.
• OptionParser: Argument parser for C++ programs.
• FileParser: Text file parser (tokenization)
• TensorAlloc: Tensor aligned memory allocation for linear algebra computing
• PhoenixProtocolBuffer: Custom optimized protocol buffer library.

Optional dependencies

• PhoenixZMQ: Helper library on top of cppzmq. Only used if zmq is found.
• PhoenixDebug: Write data to file while in debug mode. Only used if debug mode is activated
• PhoenixHardware: Detect hardware information at compilation time and profile functions. Only used when profiler mode is activated.

MacOS

Hiperta is developped for Linux. MacOS is not officially supported but HiPeRTA is nonetheless functionnal in most cases.

Installation

In $HOME

You only have to clone HiPeRTA :

  $ git clone https://gitlab.in2p3.fr/CTA-LAPP/RTA/HiPeRTA.git
  $ cd HiPeRTA
  $ ./install.sh

Then HiPeRTA is installed in your $HOME/usr.

In a conda environment

If you installed HiPeRTA's dependencies using a conda environmenent, HiPeRTA can automatically detect it and install itself inside the conda environmenent. To do so:

  $ conda activate hiperta_env
  $ git clone https://gitlab.in2p3.fr/CTA-LAPP/RTA/HiPeRTA.git
  $ cd HiPeRTA
  $ ./install.sh

Then HiPeRTA is installed in the hiperta_env conda environmenent and can be used when the environment is activated. Check the value the value of $CONDA_PREFIX in the build output to make sure the conda environment is well detected.

Install using pixi conda manager

HiPeRTA is packaged using pixi, a modern and fast conda package manager. Pixi simplifies environment management and installation, ensuring reproducible builds and easy dependency handling. Phoenix packages are hosted on prefix.dev, with two dedicated channels:

• cta-sag for stable releases.
• cta-sag-dev for development packages. These packages are build from the latest developments in the Phoenix projects, they can be deleted at any moment. Using pixi is recommended for people that get started using or developing Phoenix packages. It will speed up your setup, handle the dependencies resolution and updates, and allow you to use the tasks defined for common development operations (run test, build doc, etc.)

To install the package in a globally accessible location and exposes its command line applications:

pixi global install -c conda-forge -c https://prefix.dev/cta-sag hiperta

Using this command you can specify :

• the channel where the package is located: https://prefix.dev/cta-sag
• the version you want to install, for instance "hiperta==0.8.0"

To use this library as a dependency in your own pixi projects, you must add the dependency into your workspace. First, add the cta-sag channel into your workspace:

[workspace]
channels = [
    "https://prefix.dev/cta-sag"
]

then add the dependency on the library. For instance to add the library in your default environment:

pixi add hiperta

Custom install paths

If you prefer a customized install path you can do :

  $ git clone https://gitlab.in2p3.fr/CTA-LAPP/RTA/HiPeRTA.git
  $ cd HiPeRTA
  $ ./install.sh /your/install/path

Custom compilation options

If you prefer a customized install path with custom compilation you can do :

  $ git clone https://gitlab.in2p3.fr/CTA-LAPP/RTA/HiPeRTA.git
  $ cd HiPeRTA
  $ mkdir build && cd build
  $ cmake .. -DCMAKE_INSTALL_PREFIX=/your/install/Path -DRELEASE_MODE=yes
  $ make -j `nproc`
  $ make install -j `nproc`

The nproc gives the number of cores of the computer. If you want a build on one core you can just type :

  $ make
  $ make install

The available cmake [yes/no] options are:

• SELF_PERFORMANCE_TESTS: compile performance tests
• ENABLE_PROGRAMS: compile HiPeRTA entrypoint programs. This is the default mode when the I/O dependencies are present.
• ENABLE_ANALYSIS_TEST_PROGRAMS: compile high level tests of HiPeRTA hiperta_r0_dl1.
• ENABLE_ANALYSIS_STREAM_TEST_PROGRAMS: compile high level tests of hiperta_stream_r0_dl1 and hiperta_stream_r0.
• ENABLE_PROFILER: compile HiPeRTA with profiling enabled.

Update HiPeRTA

There is a convenience script to pull changes from remote (git pull, stays the current branch) and re-install the software:

  $ # if using conda: conda activate hiperta_env
  $ cd HiPeRTA
  $ ./update.sh

If you want to update and install in a custom path :

  $ cd HiPeRTA
  $ ./update.sh /your/install/path

Using HiPeRTA Containers

Containers are automatically build in CI/CD to provide easily deployable artifacts for HiPeRTA. They are available in LAPP registry and are also available in the CTAO repository registry here.

The images are build on top of debian-slim base images. An image is build for the head of each branch in the repository, as well as for each realease

To get the image for the release 7.0.5:

apptainer pull gitlab-registry.in2p3.fr/cta-lapp/rta/hiperta/prod:7.0.5

To get the image for a branch, the tag is build as lastRelease_branchName, for instance 7.0.5_main.

### VSCode dev containers configuration:

// For format details, see https://aka.ms/devcontainer.json. For config options, see the
// README at: https://github.com/devcontainers/templates/tree/main/src/miniconda
{
	"initializeCommand": "docker pull gitlab-registry.in2p3.fr/cta-lapp/rta/hiperta/dev:latest",
	"image": "gitlab-registry.in2p3.fr/cta-lapp/rta/hiperta/dev:latest",
	"postAttachCommand": "git config --global --add safe.directory /src && rm -rf tmp_project && rm -rf /src/build && mkdir /src/build && cd /src/build && cmake .. -DENABLE_PROGRAMS=yes -DSELF_TESTS_MODE=yes -DSELF_PERFORMANCE_TESTS=no -DENABLE_ANALYSIS_TEST_PROGRAMS=yes -DENABLE_ANALYSIS_TEST_PROGRAMS=yes -DDOC_MODE=YES && make all -j `nproc`",
	"customizations": {
		// Configure properties specific to VS Code.
		"vscode": {
			// Add the IDs of extensions you want installed when the container is created.
			"extensions": [
				"wmaurer.change-case",
				"ms-vscode.cpptools-extension-pack"
			]
		}
	},
	"workspaceMount": "source=${localWorkspaceFolder},target=/src,type=bind",
	"workspaceFolder": "/src"
}

Citation

Proceeding

To cite HiPeRTA, please refer to this proceeding

bibtex entry:

@article{Caroff:2023lgp,
    author = "Caroff, Sami and Aubert, Pierre and Garcia, Enrique and Maurin, Gilles and Pollet, Vincent and Vuillaume, Thomas",
    collaboration = "CTA LST Project",
    title = "{The Real Time Analysis framework of the Cherenkov Telescope Array's Large-Sized Telescope}",
    eprint = "2309.11679",
    archivePrefix = "arXiv",
    primaryClass = "astro-ph.IM",
    reportNumber = "CTA-ICRC/2023/41",
    doi = "10.22323/1.444.0616",
    journal = "PoS",
    volume = "ICRC2023",
    pages = "616",
    year = "2023"
}

Zenodo

Hiperta is hosted on Zenodo