Skip to content

Datasets

To ensure interoperability between components, OpenProblems uses AnnData as the standard data format for both input and output files of components, and strict requirements are imposed on the format of these files.

AnnData objects have a structured format that includes the main data matrix (X, e.g. gene expression values), annotations of observations (obs, e.g. cell metadata), annotations of variables (var, e.g. gene metadata), and unstructured annotations (uns). This organization makes it easy to work with complex datasets while maintaining data integrity and ensuring a standardized structure across different components.

File format specifications

Section titled “File format specifications”

All OpenProblems tasks contain specifications for the exact format of all H5AD inputs and outputs for all components in the workflow. These specifications describe the required and optional fields in the AnnData objects, as well as descriptions of those fields. They are used to validate the input and output files of components and to generate the documentation for the API of each component.

You can find these specifications in the src/api/file_*.yaml of each task. Here is an example: src/datasets/api/file_raw.yaml.

Common datasets

Section titled “Common datasets”

OpenProblems offers a collection of common datasets that can be used to test components and run benchmarking tasks. These datasets are generated by dataset loaders and processed by a common processing pipeline stored in src/datasets.

File format

Section titled “File format”

The format of common datasets is based on the CELLxGENE schema along with additional metadata specific to OpenProblems (in the .uns slot) and some additional output generated by our dataset preprocessors (in the .layers, .obsm, .obsp and .varm slots).

Here is what a typical common dataset looks like when printed to the console:

AnnData object
  obs: 'dataset_id', 'assay', 'assay_ontology_term_id', 'cell_type', ...
  var: 'feature_id', 'feature_name', 'soma_joinid', 'hvg', 'hvg_score'
  obsm: 'X_pca'
  obsp: 'knn_distances', 'knn_connectivities'
  varm: 'pca_loadings'
  layers: 'counts', 'normalized'
  uns: 'dataset_id', 'dataset_name', 'dataset_url', 'dataset_reference', ...

Some slots might not be available depending on the origin of the dataset. Please visit the reference documentation on the common dataset file format for more information on each slot.

Available datasets

Section titled “Available datasets”

Our datasets are stored in s3://openproblems-data/resources/datasets. Please visit the Datasets page for more information on each available dataset.