Where Open Problems is headed.
A community-driven platform grows continuously. Below is what we've shipped, what we're building now, and the directions we're exploring, separated honestly so the picture survives scrutiny.
Foundations
The project is founded; the first tasks, the Viash + Nextflow stack, and the NeurIPS 2021 multimodal competition establish the model.
- First task definitions & control methods
- NeurIPS 2021 multimodal competition
- Reproducible component framework
Scale & rigor
The platform matures into a continuously-run living benchmark with a standardized dataset pipeline and public leaderboards.
- Standardized AnnData dataset pipeline
- Continuous leaderboards on the website
- NeurIPS 2023 perturbation competition to OP3
Foundation-model benchmarking (MVP)
Bring single-cell foundation models into the platform and evaluate them on established downstream tasks.
- Integrate SCimilarity, scGPT, UCE, Geneformer & scPRINT
- Evaluate on Batch Integration, Label Projection & Perturbation Prediction
- Publish a white paper / preprint on results
Experience & infrastructure
Make the platform easier to use and to build on, and harden the core infrastructure.
- Python package for quick local evaluation
- Partial results for long-running methods; local preview of figures & QC
- Data-lineage tracking and SpatialData support
- Switch between cloud providers; DB schema refactor
Global community & impact
Grow a global community, translate benchmark insights into practice, and keep infrastructure cost-effective.
- Global meetings, onboarding & support
- Link dataset characteristics to method performance; publish guidelines
- Annual hackathon (~50 people, ~6 topics)
- API clients & a web client
Aspirational & forward-looking
Directions we're actively interested in but haven't committed to a timeline for, shaped by what new contributors bring.
Want to shape what's next?
Join a hackathon, propose a task, or contribute a method. The roadmap is set in the open.