How the application monitoring the spread of the disease was created

The “COVID-19: The spread of the disease in Greece and worldwide” was created and is managed by iMEdD Lab. Its purpose is to facilitate the work of the journalistic community and to inform anyone interested in the spread of the pandemic in Greece and around the world. 

It was first launched online on 16 March 2020 and, since then, it is constantly updated with the most recently known information. From its first launch to last November, the application had been regularly updated with various new features and additional analyses. Fully redesigned as it is today, the application was presented on December 2, 2020, when iMEdD Lab launched the version 2.0. 

The application shows the geographical distribution of confirmed recorded cases in our country and globally, and of COVID-19 fatalities by country in the world map. It also comprehensively presents analyses on the evolution of epidemiologic data in Greece and across the globe, by country. 

• Embed
  <iframe src="https://lab.imedd.org/covid19/iframe/map-view?lang=en&mapLevel=greece&period=1&mapKey=cases" height="500" width="100%" allow="fullscreen"></iframe>Αντιγραφή

The data sources

All epidemiologic data for Greece, except for the geographical distribution of deaths, are based on official information announced by the competent bodies. Since February 26, 2020, iMEdD Lab has been collecting official data from multiple channels of distribution and it maintains them in open datasets, which are available in the relevant repository, in iMEdD Lab’s account on GitHub.

Specifically, all our analyses about cases, intubated and recovered patients, the total number of deaths and samples tested in Greece derive from datasets structured by iMEdD Lab based on information released through: 

• daily epidemiological surveillance reports published by EODY in PDF format
• other daily announcements and informal updates emailed to the Press by EODY 
• briefings to accredited health journalists by the Ministry of Health
• informal updates emailed to the Press by the Ministry of Health
• updates released by the General Secretariat of Civil Protection in .pdf files (indicatively) 
• the covid19.gov.gr platform

An important exception to the above concerns the geographical distribution of deaths: competent bodies have not, as of yet, officially disclosed any relevant information. For this reason, pertinent data, which is part of iMEdD Lab’s archived datasets and was presented in the first version of the application, is a synthesis of cross-referenced information that has been published in the Greek Press. This data covers the period spanning from the onset of the pandemic in Greece until mid-November 2020. Since then, no relevant information has been published, and as a result, the geographical distribution of deaths throughout Greece is not presented in the second version of the application, and said datasets are not updated any more (see also section “Limitations of Data Analysis”).

IMEdD Lab collects the abovementioned primary information, extracts data from different file formats (PDF, .doc, body text in e-mail and .html) and converts it into structured datasets.

The source of the data regarding COVID-19 vaccination in Greece is data.gov.gr, where data is updated daily, and an API is available including data on the daily number of vaccination doses received (API key: daytotal), the cumulative number of vaccination doses received (ΑPI key: totalvaccinations), and the cumulative number of vaccinated people with at least one dose (API key: totaldistinctpersons), broken down by date and regional unit. Data have been provided since December 28, 2020. Unfortunately, a documentation on the API has not been published on data.gov.gr, but said official API keys are used by iMEdD Lab for the purposes of the analyses presented through our web application.

The global data comes from the repository of Johns Hopkins University on GitHub. In this repository, the University makes the data collected by its research team available for educational and research purposes. This database is updated once a day, as well as the relevant visualizations on iMEdD Lab’s application. That data is shown on the world map, as well as in charts presented in the “Insights on the World” section. More information about the corresponding online application of Johns Hopkins University, which was launched on 22 January 2020, its development and the data it uses can be found here.

Data processing and review

Within the context of performing data synthesis and daily updating datasets with epidemiological data on Greece, systematic review and regular processing of data is required. Some typical examples include:

• Due to a lack of daily data, it was necessary to handle missing values. With respect to tests, in particular, reports regarding the cumulative number of daily tests carried out during the first wave of the pandemic in the country were not systematically issued: there were intervals during which a shortage of published data was identified. In such cases, missing values ​​were filled in by iMEdD Lab with the daily average of the tests corresponding to the number of days that had elapsed between the last known total test number and the immediately preceding one. For example, on June 4, 2020, it was announced that a total of 193,929 clinical laboratory samples had been tested. Then, from 5 to 8 June 2020, there was no relevant information, whereas, on 9 June 2020, it was announced that a total of 231,759 samples had been tested since the onset of the pandemic. Upon calculating the daily number of new tests regarding the intervals between official reports, missing values ​​were replaced by a daily average of 37,830 tests that are said to have been performed in total during the reporting period.

• Regarding the period between the end of April 2020 and mid-June 2020, data on the geographical distribution of cases across the country was obtained from the covid19.gov.gr platform, where said data was found not to have been updated on a daily basis. For instance, on Wednesday May 13, 2020, EODY announced that, by then, a total of 2,760 cases had been recorded in the country, a number that coincided with the then reported total number of cases on the platform. As regards data on the geographical distribution of these cases – as published at covid19.gov.gr – 21 cases were labeled as “under investigation” in terms of area of reference. The following day, on Thursday 14 May 2020, EODY announced 10 new cases and a total of 2,770 confirmed cases in the country. The total number of cases was updated on the covid19.gov.gr platform, but data on the geographical distribution of cases was not revised: as a result, on the one hand, the sum of reported cases per region did not correspond to the total of 2,770 registered patients and, on the other, the number of “under investigation” cases was under-reported on the platform (21 instead of 31). This inconsistency persisted throughout the following days, up until Monday 18 May 2020, when the total number of cases on the platform (2,836) was updated, according to the daily report issued by EODY on that day, but the sum of cases per region, as presented at covid19.gov .gr, was off by 76. On November 29, 2020, the total number of cases presented on the covid19.gov.gr platform was 97,288, which corresponded to the total number of cases EODY had announced four days prior, on November 25. At the same time, the sum of cases per region, presented in the geographical distribution table on the platform, is 21,372 – nine cases have not been reported in order for the sum to correspond to the total number of cases announced by EODY on October 8, 2020. Similar inconsistencies on the platform have been detected by iMEdD Lab before, in which case the remaining cases would be marked as “No Location Provided”, both in the relevant datasets and the application.

• Embed
  <iframe src="https://lab.imedd.org/covid19/iframe/greece-key-daily-agg-bar?lang=en" height="500" width="100%" allow="fullscreen"></iframe>Αντιγραφή

• Regarding the period between the end of April 2020 and mid-June 2020, data on the geographical distribution of cases across the country was obtained from the covid19.gov.gr platform, where said data was found not to have been updated on a daily basis. For instance, on Wednesday May 13, 2020, EODY announced that, by then, a total of 2,760 cases had been recorded in the country, a number that coincided with the then reported total number of cases on the platform. As regards data on the geographical distribution of these cases – as published at covid19.gov.gr – 21 cases were labeled as “under investigation” in terms of area of reference. The following day, on Thursday 14 May 2020, EODY announced 10 new cases and a total of 2,770 confirmed cases in the country. The total number of cases was updated on the covid19.gov.gr platform, but data on the geographical distribution of cases was not revised: as a result, on the one hand, the sum of reported cases per region did not correspond to the total of 2,770 registered patients and, on the other, the number of “under investigation” cases was under-reported on the platform (21 instead of 31). This inconsistency persisted throughout the following days, up until Monday 18 May 2020, when the total number of cases on the platform (2,836) was updated, according to the daily report issued by EODY on that day, but the sum of cases per region, as presented at covid19.gov .gr, was off by 76. On November 29, 2020, the total number of cases presented on the covid19.gov.gr platform was 97,288, which corresponded to the total number of cases EODY had announced four days prior, on November 25. At the same time, the sum of cases per region, presented in the geographical distribution table on the platform, is 21,372 – nine cases have not been reported in order for the sum to correspond to the total number of cases announced by EODY on October 8, 2020. Similar inconsistencies on the platform have been detected by iMEdD Lab before, in which case the remaining cases would be marked as “No Location Provided”, both in the relevant datasets and the application.

• From 15 June 2020 onwards, the geographical distribution of cases across the country, presented both in iMEdD Lab’s application and datasets, is based on informal updates emailed to the Press by EODY on a daily basis. During this period, the alleged, new, daily cases with “No Location Provided”, as resulting from iMEdD Lab’s application and datasets, correspond to the sum of cases whose geographical area was “Under investigation” at the time when informal updates were emailed to the Press by EODY, leaving out any duplicates that were identified during data corrections performed by the Organization, as explained above. Unfortunately, competent bodies do not issue official press briefings regarding the geographical distribution of cases and, therefore, cases falling under the label of “No Location Provided” tend to increase, even if (some of) the originally under-reported cases were subsequently assigned to specific locations by the authorities.

• Data presented by iMEdD Lab concerning cases detected at “Points of Entry” as well as “Imported Cases (Detected through Voluntary Testing)” correspond to data disclosed by EODY through informal press releases, from June 15 onwards. Upon reviewing data, a discrepancy was identified between the sum of cases linked to incoming travelers and the number of cases concerning people who are not holders of permanent residency in Greece, according to covid19.gov.gr. For example, from July 1, 2020 to October 8, 2020 (when the geographical distribution of cases was last updated on the platform), the number of cases tagged as “Non-Permanent Residents of Greece” was 1,566 according to covid19.gov.gr. During the same period, the total number of cases linked to incoming travelers (“Points of Entry” and “Imported Cases”) amounted to 2,299, according to informal EODY updates. The number of cases labeled “Non-Permanent Residents of Greece” reported on covid19.gov.gr, is almost identical to data disclosed by the General Secretariat for Civil Protection, according to which, the number of confirmed cases linked to individuals entering the country was 1,557 (from 1 July to 4 October 2020). Therefore, in cases of confirmed infections initially detected at Points of Entry or through voluntary testing, which the Authorities attributed to Greek residents (and not visitors) at a later time than EODY’s initial informal press release, then these cases are not mapped by iMEdD Lab. The reason for this is that the General Secretariat for Civil Protection provided data on the geographical distribution of imported cases per Region, while EODY issues informal Press briefings on the geographical distribution of cases per Regional Unit. Therefore, in the case of the aforementioned detected discrepancy, the geographical distribution of cases is impossible unless we obtain consolidated information on each Regional Unit from competent bodies.

Data analysis and visualizations

The world map shows the total number of recorded cases and deaths per 100,000 population for each country. The map of Greece shows the total number of recorded cases per 100,000 population for each area (Attica Region, by Regional Unit for the rest of the mainland and by Prefecture for the island country): this calculation is done secondarily by iMEdD Lab, taking into account the absolute number of recorded cases and deaths, which are known per Prefecture, and the data on the resident population based on the 2011 census, as provided by ELSTAT.

Respectively, the map of Greece on the course of vaccination across the country shows the number of people who have received at least one dose of COVID-19 vaccine per 100,000 population for each regional unit in Greece. Data on the resident population based on the 2011 census, as provided by ELSTAT, are also used for this calculation.

This “normalisation of numbers” (in this case, the recording of the number of recorded cases per 1,000,000 and per 100,000 population, around the world and in the Prefectures of Greece respectively) has been selected for two reasons: on the one hand, it allows the visualisation of data on a choropleth map, such as that used by iMEdD Lab. On the other hand, it allows a more correct understanding of the issue of the spread of the virus per country or per Prefecture, allowing a comparison of the extent of the problem based on population size.

However, in reading any such map of cases, attention is brought to the fact that, for example, there are differences in terms of the laboratory tests various countries perform: Some countries opt for mass testing of the population and therefore announce more confirmed cases. Other countries conduct targeted testing on smaller populations, so they may record fewer cases than is the case. Thus, there can be significant differences in the mapping of the actual spread of the virus in each country, even between states with equal populations. For this reason, the normalisation of recorded cases based on the laboratory tests that each country has conducted would be even more appropriate. So far, the relevant publicly available official data found are not recorded in one homogeneous and standardized way for every country. For instance, a country may report the total number of samples tested, while another country might report the total number of people tested. Thus,  different policies of various countries regarding both testing and the way tests are reported might drive to underestimations or overestimations. You could visit Our world in data, if you wish to read more about testing data.

A 7-day moving average is additionally presented in every chart showing the evolution of epidemiological data over time (i.e. daily new cases or cumulative cases up to each day of reference): the 7-day moving average is based on secondary calculations, using available official data, and is used to smooth values recorded and announced by the authorities. In specific, the rolling average of each variable (cases, deaths, etc) is calculated and represented for each day, taking into account the data of the last seven days before it.

• Embed
  <iframe src="https://lab.imedd.org/covid19/iframe/greece-key-subplot-regions?lang=en" height="500" width="100%" allow="fullscreen"></iframe>Αντιγραφή

In the “Insights on Greece” section, the chart entitled “Evolution of new cases by area in Greece” represents daily new cases both per 100,000 residents and in absolute numbers, in each area. Areas are ranked based on the number of new cases per 100,000 residents, meaning that districts on top currently face a more severe outbreak. International data are similarly visualized in the chart entitled “Evolution of new cases and deaths by country” in the “Insights on the World” section.

The table entitled “Vaccination in each area in Greece” records the course of COVID-19 vaccination per regional unit in Greece. By default, it is sorted by the percentage of people that have been vaccinated in each area. The source of the vaccination data is data.gov.gr, and they are updated daily. Clicking on the field “Vaccinated per 100k” sorts the table accordingly. This figure helps us compare the total number of vaccinated people across all regional units. For instance, 1,000 vaccinated people per 100k population in Central Athens would mean that, based on the provided data, if Athens had 100,000 habitants, 1,000 of them would be vaccinated. Therefore, we can compare areas with different populations. The table is sorted by total vaccination doses received, when users click on the field “Vaccinations”.

Tools we use

Scripts developed with the use of Python programming language run for the retrieval, the extraction and the processing of data. As for the retrieval of content enclosed in the daily epidemiological surveillance report issued by EODY in .pdf files, the Apache TIKA library is used. 

In the past, one of the tools that have been used to collect global data from Worldometer is Workbench, an application that makes it easier for journalists to collect and analyse data without requiring programming language skills, has been used for the collection and the processing of global data deriving from Worldometer. Workbench was developed in Greek in the framework of the first cycle of the iMEdD Incubator.

The map background has been sponsored by the Mapbox Community team, which encloses data deriving from the open and publicly available Open Street Map.  

The application is developed with JavaScript, by Civic Information Office (CVCIO) and it is an open code project. Charts are created with the use of D3.js. You may find more information about scripts developed, tools and libraries used on the relevant GitHub repository. 

Plotly Python Open Source Graphic Library was also used to create charts in the “STATISTICS” section of the former version 1.0 of the application.

Limitations of data analysis

Currently, the biggest constraint the application faces relates to data consistency, due to a lack of official data on the geographical distribution of COVID-19 deaths in the country. Specifically, data on the geographical spread of fatalities, which was presented in the first version of the application and is currently stored in iMEdD Lab’s datasets for archival reasons, was obtained from press reports on patient deaths. However, since mid-November, the flow of relevant information has been halted, while no competent body has, as of yet, officially disclosed information as to which areas or hospitals each new death is recorded in.

For this reason, the new daily fatalities are considered as deaths with “No Location Provided” by iMEdD Lab, until further information on the geographical distribution of deaths is made public. Until then, map visualization of deaths by region is no longer possible. To cite one example, on November 9, 2020, the estimated number of deaths falling under the “No Location Provided” category was 133, out of a total of 830. As of November 30, 2020, there were 1,641 geographically unspecified deaths, out of a total of 2,406 deaths that have been recorded since the onset of the pandemic in the country. This means that, over a period of 21 days, the rate of deaths in unspecified locations increased from 16% to 68%. Essentially, the dramatic increase in lack of information on the geographical distribution of coronavirus fatalities in the country coincided with the spike in deaths, considering that 74% of deaths recorded from the onset of the pandemic to November 30 in Greece occurred in November – on October 31, the country saw a total of 626 COVID-19 related deaths, while by November 30 fatalities had increased to 2,406.

Over time, iMEdD Lab’s desired outcome is the visualization of the distribution of epidemiological data by more detailed geographical level (for example, by municipality) on the map of Greece. At the moment, this is not possible, since relevant data is not being published by competent bodies.