Skip to content

Create a dataset loader

A dataset loader generates one or more raw datasets. A raw dataset is processed by the dataset preprocessing workflow to create a common dataset that can be used by multiple benchmarking tasks.

This guide shows how to create a new Viash component to fetch or generate datasets.

OpenProblems datasets

Section titled “OpenProblems datasets”

Common datasets are created by generating raw datasets with a data loader and running them through the pre-processing pipeline. Afterwards, further task-specific processing occurs prior to the task-specific benchmarking workflow.

Flow of data in OpenProblems benchmarks. Legend: grey rectangles are AnnData (.h5ad) files; purple parallelograms are Viash components.

DatasetloaderRawdatasetPre-processingCommondatasetBenchmarkingworkflowResults
DatasetloaderRawdatasetPre-processingCommondatasetBenchmarkingworkflowResults
Flow of data in OpenProblems benchmarks. Legend: grey rectangles are AnnData (.h5ad) files; purple parallelograms are Viash components.

Step 1: Create a directory for the dataset loader

Section titled “Step 1: Create a directory for the dataset loader”

To add a dataset to OpenProblems, create a Viash component for a dataset loader.

mkdir src/datasets/loaders/myloader

Step 2: Create a Viash config

Section titled “Step 2: Create a Viash config”

Create a config for the dataset loader. The Viash config contains the component’s metadata, which script runs it, and its required dependencies.

src/datasets/loaders/myloader/config.vsh.yaml
functionality:
  name: "myloader"
  namespace: "datasets/loaders"
  description: "A new dataset loader"
  arguments:
    - name: "--output"
      __merge__: ../../api/file_raw.yaml
      direction: "output"
  resources:
    - type: python_script
      path: script.py
platforms:
  - type: docker
    image: python:3.10
    setup:
      - type: python
        pypi: anndata~=0.8.0
  - type: nextflow

For additional parameter options, see the Parameters section.

Step 3: Create a script

Section titled “Step 3: Create a script”

Create a script that generates or loads the dataset. The example below generates a random dataset; check src/datasets/loaders for real data examples. The output AnnData object must follow the format described below.

src/datasets/loaders/myloader/script.py
import anndata as ad
import pandas as pd
import scipy
import numpy as np
import random


## VIASH START
par = {"output": "output.h5ad"}
## VIASH END


obs = pd.DataFrame({
    "cell_type": random.choices(["enterocyte", "intestine goblet cell", "stem cell"], k=100),
    "batch": random.choices(["experiment1", "experiment2"], k=100),
    "tissue": random.choices(["colon", "ileum"], k=100)
})


var = pd.DataFrame(index=["APP", "AXL", "ADA", "AMH"])


counts = scipy.sparse.csr_matrix(
    np.random.poisson(0.3, (obs.shape[0], var.shape[0]))
)


adata = ad.AnnData(
    layers={"counts": counts},
    obs=obs,
    var=var,
    uns={
        "dataset_id": "my_dataset",
        "dataset_name": "My dataset",
        "data_url": "https://url.to/dataset/source",
        "data_reference": "mydatasetbibtexreference",
        "dataset_summary": "A short description of the dataset.",
        "dataset_description": "Long description of the dataset.",
        "dataset_organism": "homo_sapiens"
    }
)


adata.write_h5ad(par["output"], compression="gzip")

Step 4: Run the component

Section titled “Step 4: Run the component”

Rebuild the Docker container after changing the platforms section:

viash run src/datasets/loaders/myloader/config.vsh.yaml -- ---setup cachedbuild

Run the component:

viash run src/datasets/loaders/myloader/config.vsh.yaml -- --output mydataset.h5ad

Parameters

Section titled “Parameters”

Add arguments to the dataset loader by extending the functionality.arguments section:

arguments:
  - name: "--n_obs"
    type: "integer"
    description: "Number of cells to generate."
    default: 100
  - name: "--n_vars"
    type: "integer"
    description: "Number of genes to generate."
    default: 100

Access them via par in the script:

obs = pd.DataFrame({
    "cell_type": random.choices(["enterocyte", "intestine goblet cell", "stem cell"], k=par["n_obs"]),
    "batch":     random.choices(["experiment1", "experiment2"], k=par["n_obs"]),
    "tissue":    random.choices(["colon", "ileum"], k=par["n_obs"])
})
var = pd.DataFrame(index=[f"Gene_{i}" for i in range(par["n_vars"])])

Format of a raw dataset object

Section titled “Format of a raw dataset object”

The AnnData output must contain at least the following slots (derived from the CELLxGENE schema v4.0.0). The authoritative source is src/api/file_raw.yaml in the datasets repository.