The overall data is available in two states: as raw data and/or as pre-processed data. Additionally there are three reference tables for variable lookup.

Reference tables

Raw data

The raw data was only processed if this was necessary for patient de-identification and otherwise left unchanged compared to the original source. The raw data contains the complete set of available variables (681 variables). It consists of the following tables:

Pre-processed data

The pre-processed data consists of intermediary pipeline stages from our original Nature medicine publication. Source variables representing the same clinical concepts were merged into one meta-variable per concept. The data contains the 18 most predictive meta-variables only, as defined in our publication. Two different stages of the pipeline are available

The code used to generate these stages can be found in this GitHub repo under the preprocessing folder.

Which data to use?

The pre-processed data is intended mainly as a quick way to jump-start a project or for use in a proof of concept. We recommend using the source data whenever possible for regular projects. It is the most flexible form and contains the complete set of variables in the original time resolution.

Data formats

Data is available in two formats: CSV for wide compatibility and Apache Parquet for convenience and performance. Parquet is a strongly typed, binary format that is supported by many major data processing tools such as pandas, spark, R, matlab, etc.