COVID-19 Integrated Surveillance Data in Italy
COVID-19 integrated surveillance data provided by the Italian National Institute of Health and processed via UnrollingAverages.jl to deconvolve the weekly simple moving averages.
## Overview Every week the National Institute for Nuclear Physics ([INFN](https://home.infn.it/it/)) imports an anonymous individual-level dataset from the Italian National Institute of Health ([ISS](https://www.iss.it)) and converts it into an incidence time series data organized by date of event and disaggregated by sex, age and administrative level with a consolidation period of approximately two weeks. The information available to the [INFN](https://home.infn.it/it/) is summarised in the following [meta-table](https://covid19.infn.it/iss/campi-iss.pdf). ## Data ### Archive The original [data](https://covid19.infn.it/iss/) has been stored [here](https://github.com/InPhyT/COVID19-Italy-Integrated-Surveillance-Data/tree/main/0_archive), reorganised [here](https://github.com/InPhyT/COVID19-Italy-Integrated-Surveillance-Data/tree/main/1_structured_archive). ### Input The input data has been stored [here](https://github.com/InPhyT/COVID19-Italy-Integrated-Surveillance-Data/tree/main/2_input) and contain the following information: - Aggregated data in the [`daily_incidences_by_region`](https://github.com/InPhyT/COVID19-Italy-Integrated-Surveillance-Data/tree/main/2_input/daily_incidences_by_region) folder: * Weekly moving average and daily time series of **confirmed cases by date of diagnosis** at the regional level; * Weekly moving average and daily time series of **ordinary hospital admissions by date of admission** at the regional level; * Weekly moving average and daily time series of **intensive hospital admissions by date of admission** at regional level; * Weekly moving average and daily time series of **deceased cases by date of death** at the regional level. - Disaggregated data in the [`daily_incidences_by_region_sex_age`](https://github.com/InPhyT/COVID19-Italy-Integrated-Surveillance-Data/tree/main/2_input/daily_incidences_by_region_sex_age) folder: - Weekly moving average time series of **symptomatic cases by date of symptoms onset** stratified by sex and age at the regional level; - Weekly moving average time series of **confirmed cases by date of diagnosis** stratified by sex and age at the regional level; - Weekly moving average time series of **ordinary hospital admissions by date of admission** stratified by sex and age at the regional level; - Weekly moving average time series of **intensive hospital admissions by date of admission** stratified by sex and age at the regional level; - Weekly moving average time series of **deceased cases by date of death** stratified by sex and age at the regional level. ### Output The output data has been stored [here](https://github.com/InPhyT/COVID19-Italy-Integrated-Surveillance-Data/tree/main/3_output/data) and contain the following information: * Reconstructed daily time series of **confirmed cases by date of diagnosis** stratified by sex and age at the regional level; * Reconstructed daily time series of **symptomatic cases by date of symptoms onset** stratified by sex and age at the regional level; * Reconstructed daily time series of **ordinary hospital admissions** by date of admission stratified by sex and age at the regional level; * Reconstructed daily time series of **intensive hospital admissions** by date of admission stratified by sex and age at the regional level; * Reconstructed daily time series of **deceased cases by date of death** stratified by sex and age at the regional level. ## Methodology ### Data Organization Raw data are downloaded from the [INFN](https://covid19.infn.it/iss/) (direct download [here](https://covid19.infn.it/iss/csv_part/iss.tar.gz)), decompressed, stored in the [`0_archive`](https://github.com/InPhyT/COVID19-Italy-Integrated-Surveillance-Data/tree/main/0_archive) folder and then organized into the [`1_structured_archive`](https://github.com/InPhyT/COVID19-Italy-Integrated-Surveillance-Data/tree/main/1_structured_archive) folder via the execution of the [data_organization.jl](https://github.com/InPhyT/COVID19-Italy-Integrated-Surveillance-Data/blob/main/src/data_organization.jl) script. ### Data Processing In general, given the moving average (or rolling mean) of a time series, it's not possible to recover the original series unless *n* original points are known where *n* is the width of the window adopted in the moving average, but since epidemiological surveillance incidence series are strictly composed of natural numbers, we can leverage this property to come up with a finite number of candidate original series, and then prune these down to as little as possible, hopefully only one, final recovered series. The whole procedure is performed via the execution of the [main.jl](https://github.com/InPhyT/COVID19-Italy-Integrated-Surveillance-Data/blob/main/src/main.jl) script and the related technical details can be found the documentation of [UnrollingAverages.jl](https://github.com/InPhyT/UnrollingAverages.jl) package. The averaged time series to be **unrolled** (i.e. recovered, reconstructed or de-averaged) are those stored in the [`2_input/daily_incidences_by_region_sex_age`](https://github.com/COVID19-Italy-Integrated-Surveillance-Data/tree/main/2_input/daily_incidences_by_region_sex_age) folder: they are organized in .csv files, each of which reporting the 10 age-specific time series of a particular incidence in a particular region. Each dataset has two counterparts that are further stratified by sex. Since the smaller the numbers involved the better UnrollingAverages.jl seems to perform, we opted for unrolling the sex-stratified series first and then aggregate them later. Since not all the age and sex stratified averaged series allows UnrollingAverages.jl to find an unique original series and no further sex-stratified information is provided by INFN, we attempted to directly unroll the sex-aggregated time series for which CovidStat provides additional information in the form of age-aggregated original time series, that we employed to select that combination of age-disaggregated series proposed by UnrollingAverages.jl which summed to the age-aggregated original time series provided by INFN. The utilized age and sex-aggregated may be found in the [`2_input/daily_incidences_by_region`](https://github.com/InPhyT/COVID19-Italy-Integrated-Surveillance-Data/tree/main/2_input/daily_incidences_by_region) folder. We'll refer to the last selection algorithm as the **cross-sectional consistency constraint**. The successfully reconstructed time series are then saved in the [`3_output/data`](https://github.com/InPhyT/COVID19-Italy-Integrated-Surveillance-Data/tree/main/3_output/data) folder (both aggregated and disaggregated by sex), while the visualisations of those that are age-stratified and sex-aggregated may be found in [`3_output/figures`](https://github.com/InPhyT/COVID19-Italy-Integrated-Surveillance-Data/tree/main/3_output/figures). ## How to Cite If you use these data in your work, please cite this repository using the metadata in [`CITATION.bib`](https://github.com/InPhyT/COVID19-Italy-Integrated-Surveillance-Data/blob/main/CITATION.bib).