We are trying to design a data repository for storing datasets, from many different sources, in such a way that they can easily be used in online data presentations.
This data repository will be part of a website that reviews, interprets and discusses scientific reports and data on public health. It is targeted at scientists, medical experts and policy makers.
The data that we use, is collected from other organizations, in the form of database dumps and ASCII files, or sometimes simply copied from scientific studies. We are considering to link to open data sources, in the future.
Previously, we have created a generic data model in which each data set was stored in the same generic format: a data set was linked to a variable and for each measure the following dimensions were available: period, area, age, sex and social economic status, all of them optionally present in a data set. Finally, the measures were stored as floating point numbers. As an example, think about mortality of coronary heart diseases. A record in the dataset would be the absolute number of deaths in the year 2014 for Dutch males in the age group 65 to 75. Meta data describing the complete set, certain domain values of a domain or individual records were supported. Domain values, like 2011, male and 65-75 were shared between data sets. This allowed for searching across data sets and, to some extent, facilitated combining different data sets in one visualization. This data model, however, required extensive preparation of data before it could be imported, including un-pivoting and mapping of domain values.
Another aspect was versioning of data sets. In some cases, the most recent available data should be used, in other cases, a fixed set is to be used, even if more recent data is loaded into the repository. In that case, each visualization is linked to a specific version. This version is only upgraded if the text accompanying the visualization is reviewed and updated as well. Imagine a trend chart in which the latest years show some new trend. This newer data is to be shown only when the editor has done an analysis on that trend.
So, this is not so much about large data sets, but rather about large numbers of diverse data sets. Characteristics:
- The presence of various measures. Sometimes, more than one measure was present, for instance an absolute number as well as a percentages or a standardized rate or confidance intervals.
- Various aggreagation levels in the dimensions: years, months, days;
5-year age groups or 0-65/65+, etc.
- Meta data and footnotes on sets, dimensions and/or single values. It must be possible to show these in tabular presentations with footnote symbols linking to descriptions.
- Versioning of data sets, where newer versions remove, correct and/or add values.
- Aggregation may or may not be trivial. For absolute numbers,
aggregation is a summation. This is easily performed and will be
supported. For standardized rates, aggregation is complex and outside the scope of the repository. In such case, pre-aggregated totals must be added to the dataset, to be able to show marginals in cross-table presentations. These totals must be distinguishable from base
numbers, as it must be possible to leave them out of visualizations. Imagine a stacked bar chart over years, where the series are age
groups. A total over all ages must not be included in the stacked
- We must be able to back-track data used in presentations to the
original source, to be able to explain were they came from. So
ideally, nowhere in the work-flow, data has to be copied and pasted nor edited manually.
On top of the data repository, there will be a presentation composer. This composer must be capable of querying the repository. These queries should ideally be generated from choices the editor makes: filtering, selecting dimensions as x-axis, series, etc. Linking to a newer version of a data set should be supported, without completely redefining the presentation.
At this point, we consider supporting various of the larger datasets with a dedicated data model, and some sort of adaptors for the presentation composer, mainly to avoid various issues with importing data sets in a generic model.
Does anybody recognize this scenario and have some suggestions about best practices or maybe even an out-of-the box solution?